A method and equipment for in-situ underwater detection of unlabeled fish samples

Abstract

This invention discloses a method and device for detecting unlabeled fish samples in situ underwater. The method includes acquiring situ images captured by a monocular camera on an underwater observation platform and constructing an image-text structured knowledge base; extracting training and validation datasets for a detection model from the knowledge base and dividing the datasets into closed-set categories and open-world categories; receiving natural language text instructions from a user describing the dataset to be detected and generating text feature vectors for the target categories; training a targeted auxiliary model through a visual-text semantic space, dynamically and interactively fusing the visual features of the situ images and the text features of the categories, and performing end-to-end joint optimization of the target model; and using the optimized target model to perform unlabeled detection of fish appearing in the underwater situ images. This invention can effectively identify new species not seen during training, can perform cross-domain inference on unlabeled observation samples, and reduces retraining and maintenance costs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of general fish target detection, and particularly relates to an underwater in-situ fish label-free sample detection method and device. BACKGROUND

[0002] As the largest ecosystem on Earth, the health and stability of marine biodiversity are crucial for global climate regulation and human sustainable development. Fish, as a key component of marine ecosystems, their population changes are important indicators for assessing the distribution of marine fish resources and environmental health. Therefore, automatic and accurate detection of underwater fish is a key technology to realize smart fisheries, marine ecological monitoring and biodiversity protection and other applications.

[0003] Traditional fish survey methods, such as catch survey method, are complex to operate, and the sampling method is difficult to be scientific, and is limited by the survey conditions. This method has poor timeliness and large error. In recent years, with the popularity of image and video collection platforms such as seafloor observation networks, remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs), automated detection technology based on computer vision has been widely used in underwater target detection tasks. In deep water aquaculture net cages and open natural waters, underwater in-situ observation relies on fixed underwater platforms to carry out real-time shooting and monitoring, which is less affected by underwater optical attenuation and suspended matter interference than mobile observation. And by laying a three-dimensional observation network composed of underwater camera equipment and sensor nodes, multi-dimensional spatiotemporal data of fish activity can be continuously obtained, and the non-intrusive nature effectively reduces the interference to biological behavior, which has important application value for species statistics and downstream ecological monitoring.

[0004] Target detection algorithms based on deep learning have achieved remarkable success in specific scenarios, but these algorithms usually rely on the setting of closed-set vocabulary. This means that the model can only recognize and detect a limited number of fish classes predefined in the training phase.

[0005] This closed-set paradigm faces serious challenges when applied to complex and variable underwater in-situ environments:

[0006] 1. Poor class scalability and high cost: There are tens of thousands of fish species in the world. Whenever a new fish species needs to be detected, traditional supervised learning methods require collecting a large number of images of the new fish species and performing fine-grained manual labeling, and then retraining or incrementally training the entire model. This process not only has high cost and long cycle, but also as the number of classes increases, the difficulty of training and maintaining the model increases exponentially.

[0007] 2. Zero-shot detection capability is missing: Traditional models cannot recognize and locate fish species that have never appeared in the training dataset or extremely rare fish species, i.e., they do not have "zero-shot" detection capability. This greatly limits their application in biodiversity census, endangered species search and other exploratory tasks that need to handle open and unknown categories.

[0008] 3. Inflexible human-computer interaction: Traditional detection systems usually output preset category labels (such as "category_id_5"). Users cannot query the target of interest through natural and flexible language description, for example, they cannot directly issue instructions such as "detect all yellow and striped fish" or "find a peacock fish" to the system.

[0009] In recent years, large-scale vision-language pre-training models (VLMs), such as CLIP, have successfully aligned image visual features and text semantic features to a unified feature space through contrastive learning on massive image-text data. This cross-modal alignment capability provides a new approach to solving the above open vocabulary detection problem, but how to effectively apply it to the underwater in-situ scene with strong domain characteristics and extend from image-level classification capability to target-level precise positioning is a key problem faced by current technology development. SUMMARY

[0010] The purpose of the present application is to overcome the shortcomings of the prior art and provide a method and device for detecting unlabeled samples of fish in situ underwater.

[0011] To achieve the above-mentioned purpose, in a first aspect, the present application provides a method for detecting unlabeled samples of fish in situ underwater, comprising:

[0012] Step 1: Obtain in-situ images taken by a monocular camera of an underwater observation platform, and based on prior knowledge in the fishing industry, decouple the fish species appearing in the images into multi-level classification labels and multiple combinations of visual attribute labels, and construct an image-text structured knowledge base;

[0013] Step 2: Extract training and validation datasets for the detection model from the knowledge base, and divide the datasets into closed-class categories and open-world categories;

[0014] Step 3: Receive a natural language text instruction input by a user to describe the dataset to be detected, and use a text encoder of a vision-language model to encode the natural language text instruction to generate a text feature vector of the target category;

[0015] Step 4, training the visual-text semantic space feature alignment of the graph-text pair dataset using a pre-trained visual-linguistic model to obtain a targeted auxiliary model;

[0016] Step 5, constructing a collaborative architecture including the target model to be optimized and the regularization reference of the targeted auxiliary model; dynamically and interactively fusing the visual features and the text features of the in-situ image; based on the structured knowledge guided multi-task learning module and the guided feature space regularization loss function of the targeted auxiliary model, a composite loss is formed to jointly optimize the target model end-to-end;

[0017] Step 6, using the optimized target model to detect fish in the underwater in-situ image without labels, locating the positions of the closed-set fish targets and open-world fish targets and classifying them.

[0018] Further, the multi-level classification identifier includes a main classification identifier and a set of hierarchical classification identifiers, the main classification identifier is used to uniquely identify the category number and standard name of the fish class, and the hierarchical classification identifier is used to represent the field of biological hierarchical relationship, including the number and name of the order and family, and the visual attribute identifier is used to describe the attribute set of observable morphological characteristics.

[0019] Further, the knowledge base includes an image sub-base and a text sub-base, the image sub-base is used to index and register the metadata of the in-situ image, and the text sub-base is used to provide a structured entry for each class in the fish dataset.

[0020] Further, the step 3 includes:

[0021] the class name in the target field sequentially embedding one or more pre-defined static text prompt templates to obtain a set of text sequences inputting each text sequence into the text encoder of the visual-linguistic model to obtain a corresponding set of text feature vectors .

[0022] Further, the step 4 specifically includes:

[0023] Step 4.1, initializing a pre-trained visual-linguistic model composed of an image encoder and a text encoder as a base model;

[0024] Step 4.2, using the closed-set class dataset and the text feature vectors of the target class to form a graph-text pair for model training;

[0025] Step 4.3, in the visual-text semantic space training, the weights of the base model are optimized by contrastive learning on the image-text pairs, maximizing the similarity between matched image visual features and text features, and minimizing the similarity between unmatched image visual features and text features, to learn the correspondence between visual concepts and text descriptions in the underwater fish field, and the trained model is used as a targeted auxiliary model.

[0026] Further, the step 5 specifically includes:

[0027] Step 5.1, the architecture of the target model is built with a visual-language model and a two-stage target detection framework, to generate candidate regions using the region proposal network in the target detection framework.

[0028] Step 5.2, the visual features and class text features of the candidate regions of the in-situ image are dynamically interactively fused using a dynamic language perception module.

[0029] Step 5.3, a structured knowledge guided multi-task learning module is used to calculate the field task loss.

[0030] Step 5.4, the feature space regularization loss is calculated by calculating the distance between the visual features extracted by the target model and the targeted auxiliary model for the same input image.

[0031] Step 5.5, the composite loss is calculated based on the field task loss and the feature space regularization loss as follows:

[0032] ;

[0033] Wherein, is the calculated composite loss, is the field task loss, is the feature space regularization loss, and λ is the weighting coefficient.

[0034] Further, the step 5.2 specifically includes:

[0035] For each class c in the structured knowledge base, a dynamic prompt Prompt_c embedded with a learnable continuous vector V is constructed, and all the dynamic prompts corresponding to the classes are input into the text encoder of the target model to generate a set of dynamic text features as text prototypes For the i-th candidate region feature A candidate region level cross-attention module is constructed:

[0036] ;

[0037] Wherein, Query vector for visual feature, 、 、 learned weight matrix, 、 Key and Value vector for text feature respectively;

[0038] The attention weight of the candidate region-level cross-attention module is represented as:

[0039] ;

[0040] wherein, is a normalized exponential function, is a transpose symbol of a matrix, is the dimension of the key vector;

[0041] The fusion feature output by the candidate region-level cross-attention module is represented as:

[0042] .

[0043] Further, the step 5.3 specifically comprises:

[0044] Taking the candidate region feature of the target model or a linear projection thereof, the text prototype of the kth class is , the temperature is , the cosine similarity between the candidate region feature of the target model and the text prototype of the kth class is is calculated as:

[0045] ; wherein,

[0046] is a similarity calculation function; The hard negative sample is used to select the most similar top-K negative class set

[0047] for each , and the main class contrast loss is calculated as:

[0048] ; wherein,

[0049] is the scaled cosine similarity between the ith candidate region and the text prototype of its positive class , and is the total number of candidate regions;

[0050] ​The classification results of the genus, species and family levels are supervised respectively by using a weighted sum of multiple cross-entropy loss terms to obtain the predicted distribution and the real label of each level , and the cross-entropy is used to calculate the level loss as follows:

[0051] ;

[0052] wherein, is a set of levels, is a weight of different levels, is a cross-entropy loss function representing a standard;

[0053] The multiple visual attributes of the target are simultaneously predicted by using a multi-label binary cross-entropy loss to obtain the predicted label and the real label of the mth attribute of the ith candidate region, and the attribute loss is calculated as follows:

[0054] ;

[0055] wherein, is the total number of attributes, is a sigmoid activation function;

[0056] The field task loss is calculated as follows:

[0057] ;

[0058] wherein, is a detection loss, , , , are weight coefficients of the main class contrast loss , the level loss , the attribute loss and the detection loss respectively.

[0059] Further, the feature space regularization loss is calculated as follows:

[0060] ;

[0061] wherein, represents a feature vector of the ith candidate region extracted by the target model, represents a feature vector of the ith candidate region extracted by the targeting auxiliary model, is an L2 norm symbol.

[0062] In a second aspect, the present application provides an underwater in-situ fish unlabeled sample detection device, comprising a storage medium and a processor, the storage medium stores a computer program, and the computer program is executed by the processor to implement the above method.

[0063] Beneficial effects: 1. The present application aligns regional-level visual features with decomposable text semantics through the synergistic effect of structured knowledge guidance and dynamic language perception modules; in the reasoning stage, new class text prototypes can be loaded on demand on unlabeled in-situ observation images / video frames and participate in regional-level similarity calculation, without the need for retraining or changing the detection network structure, thereby realizing matching detection of unseen class names or new attribute combinations in the training stage;

[0064] 2. The present application effectively solves the catastrophic forgetting problem that exists in the field of fine-tuning in the prior art. While improving the discriminability of visible classes, it retains the general semantic structure from large-scale pre-training, which helps to maintain the recognition ability of unknown classes (classes not appearing in the training stage). The resulting model has strong open set detection capability and can effectively identify new species not seen during training; at the same time, the model can successfully transfer knowledge to new application scenarios and implement cross-domain reasoning on unlabeled observation samples, thereby reducing retraining and maintenance costs. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a flowchart of the underwater in-situ fish unlabeled sample detection method of the present application embodiment;

[0066] Figure 2 is a schematic diagram of the calculation process of the composite loss of the present application embodiment. DETAILED DESCRIPTION

[0067] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, which are implemented on the premise of the technical solutions of the present application, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0068] As shown in Figure 1 and Figure 2 , the present application embodiment provides an underwater in-situ fish unlabeled sample detection method, comprising:

[0069] Step 1, structured knowledge modeling: acquire in-situ images taken by monocular cameras of underwater observation platforms, and based on prior knowledge in the field of fisheries, decouple the fish classes appearing in the images into multi-level classification labels and multiple combinations of visual attribute labels, and construct an image-text structured knowledge base. Specifically, acquire in-situ observation video sequences taken by monocular cameras of underwater observation platforms, extract representative images from the video sequences, and based on prior knowledge in the field of fisheries, decouple the fish classes appearing in the images into a main class label, a set of hierarchical classification labels, and a set of shared and combinable visual attribute labels, and construct an image-text structured knowledge base. The structured knowledge base K is a pre-constructed database stored in JSON format, including an image sub-base and a text sub-base.

[0070] The image sub-base is used to index and register metadata for in-situ images taken by monocular cameras of underwater observation platforms, and the metadata includes at least image identification, acquisition time, acquisition device information, etc. The text sub-base provides a structured entry for each class c in the fish dataset. The main class label class_name is used to uniquely identify the class number and standard name (preferably at the species level) of the object class, such as the scientific name; the hierarchical classification label is used to represent the field of biological hierarchical relationship, at least including the number and name of order and family (optionally including extended levels such as genus / subfamily); the visual attribute label is used to describe the attribute set of observable morphological features, at least including overall shape, main / secondary color, texture / pattern, key parts (such as dorsal fin / tail type), etc.

[0071] Step 2, create open vocabulary object detection dataset: extract training and validation datasets for detection models from the knowledge base, and divide the dataset into closed class and open world class. Specifically, extract representative images from the structured knowledge base image sub-base; extract 8000 frames of underwater in-situ images by manual extraction; use annotation tools to manually annotate the images, and divide the training set and validation set in the ratio of 8:2; divide the dataset again according to the open vocabulary setting to obtain Base class (closed class) and Novel class (open world class), wherein the Novel class does not participate in the training of the detection model, and is only used for zero-shot and / or open set evaluation in the validation / test phase.

[0072] Step 3, concept text instruction encoding: receive user input of natural language text instructions describing the detection dataset, and use the text encoder of the visual-language model to encode the natural language text instructions to generate text feature vectors of the target class. Specifically, the class names in the target field are sequentially embedding one or more predefined static text prompt templates, such as an English template "A photo of a [class_name]" (a photo of [class name]), to obtain a set of text sequences , inputting each text sequence into a text encoder of the vision-language model to obtain a corresponding set of text feature vectors . In addition, when multiple templates are used for the same category, the category text prototype can be taken as :

[0073] ;

[0074] Step 4, Targeted auxiliary model construction: using a pre-trained vision-language model to perform vision-text semantic space feature alignment training on the graph-text pair dataset to obtain a targeted auxiliary model. Specifically, the following steps are included:

[0075] Step 4.1, initialize a pre-trained vision-language model composed of an image encoder and a text encoder as a base model. The image encoder uses a ResNet-50 network structure, and the text encoder uses a Transformer structure.

[0076] Step 4.2, use the dataset of closed set categories divided in step 2 and the text feature vectors of the target categories generated in step 3 to form the graph-text pairs (Image-Text Pairs) for model training.

[0077] Step 4.3, in the vision-text semantic space training, the weights of the base model are optimized by contrastive learning on the graph-text pairs, maximizing the similarity between matching image visual features and text features, and minimizing the similarity between non-matching image visual features and text features, to learn the correspondence between visual concepts and text descriptions in the underwater fish domain. The trained model is used as a targeted auxiliary model for subsequent dynamic vision-language interactive training. The above scheme can effectively solve the Base-Novel Trade-off (BNT) problem. The stochastic gradient descent (SGD) optimizer is used for parameter update during model training, which is a preliminary and light fine-tuning. In the zero-shot detection setting, the matching probability of the visual feature and the text feature of each category can be calculated as follows:

[0078] ;

[0079] wherein, is the calculated matching probability, is the natural exponential function, is a similarity calculation function, is a visual feature of an input image or a candidate region.

[0080] Step 5, construct a collaborative architecture of a target model to be optimized and a targeting auxiliary model of a regularization reference; dynamically and interactively fuse the visual features of the in-situ image and the text features of the category; based on the structured knowledge guided multi-task learning module and the feature space regularization loss function guided by the targeting auxiliary model, jointly optimize the target model in an end-to-end manner. Specifically as follows:

[0081] Step 5.1, the architecture of the target model is constructed by a visual-linguistic model (CLIP) and a two-stage target detection framework (Faster R-CNN). The targeting auxiliary model is the visual-linguistic model trained in step 4 and the parameters of which are frozen in this step. The target model to be optimized is a visual-linguistic target detection model with updateable parameters. Specifically, the architecture can generate candidate regions (ROIs) by using the region proposal network (RPN) in the detection framework, and modify the recognition head into a visual-linguistic alignment module capable of conducting contrastive learning of region-level visual features ( ) and text features ( ).

[0082] Step 5.2, design a dynamic language perception module and use it to dynamically and interactively fuse the visual features of the candidate regions of the in-situ image and the text features of the category. Step 5.2 specifically includes:

[0083] For each category c in the structured knowledge base, construct a dynamic prompt Prompt_c embedded with a learnable continuous vector V, for example: Prompt_c = { [V_context], "a photo of", [V_cls], V_cls, "a fish from the", [V_fam], V_fam, "family."}. Where [V_context], [V_cls], [V_fam] are continuous vectors optimized together with other parameters of the model in the word embedding space. Input all the dynamic prompts corresponding to the categories into the text encoder of the target model to generate a set of dynamic text features as text prototypes, which contains species prototypes, attribute prototypes and superordinate prototypes.

[0084] To realize the deep alignment of regional-level visual features and text semantics, for the i-th candidate region feature A candidate region-level cross-attention module is constructed:

[0085]

[0086] wherein, is the Query vector of the visual feature, , , is the learned weight matrix, , are the Key and Value vectors of the text feature, respectively.

[0087] The attention weight of the candidate region-level cross-attention module is represented as:

[0088]

[0089] wherein, is the normalized exponential function, is the transpose symbol of the matrix, is the dimension of the key vector.

[0090] The fusion feature output by the candidate region-level cross-attention module is represented as:

[0091]

[0092] In a preferred embodiment, the hierarchical prior mask (masking non-ancestor / irrelevant prototypes) can be imposed on the Key set according to the biological hierarchical information, and the Top-K gate can be enabled to screen the text prototypes participating in the attention calculation, thereby improving the training and inference efficiency.

[0093] Step 5.3, designing a structured knowledge guided multi-task learning module, and calculating the field task loss based on the structured knowledge guided multi-task learning module. Step 5.3 specifically includes:

[0094] a) main class contrast loss : the cross-modal InfoNCE loss function is adopted, which aims to shorten the distance between the ROI feature and its positive text prototype in the feature space, while increasing the distance between the ROI feature and its negative text prototype. In a preferred embodiment, to improve the model's ability to recognize fine-grained classes, negative samples can be preferentially sampled from the same department or neighboring classes of the positive class (i.e., difficult negative sample mining). Take the candidate region feature of the target model or its linear projection, the text prototype of the k-th class is , and the temperature is ​​​, calculate the candidate region features of the target model cosine similarity with the text prototype of the kth category

[0095]

[0096] wherein, is a similarity calculation function.

[0097] Hard-Negatives are used to select the top-K negative classes for each , calculate the main class contrastive loss

[0098]

[0099] wherein, is the scaled cosine similarity between the ith candidate region and the text prototype of its positive class, is the total number of candidate regions.

[0100] b) Hierarchical loss : a weighted sum of multiple cross-entropy loss terms is used to supervise the classification results at the genus, species and family levels, to obtain the predicted distribution and the true label for each level , and the cross-entropy is used to calculate the hierarchical loss

[0101]

[0102] wherein, is the set of hierarchical levels, , is the weight of different levels, which is used to reflect the importance of different levels (for example ), is the cross-entropy loss function representing the standard. In a further preferred embodiment, to impose a higher penalty on the cross-level error on the classification level, a path consistency constraint can be added, for example, by KL divergence regularization, to constrain the consistency between the predicted hierarchical path distribution and the true hierarchical path distribution.

[0103] c) Attribute loss : a multi-label binary cross-entropy loss (BCE-with-logits Loss) is used to simultaneously predict multiple visual attributes of the target, to obtain the predicted label and the true label of the mth attribute of the ith candidate region​​​​​​​​​ and the property loss is calculated as:

[0104] ;

[0105] wherein, is the total number of properties, is a sigmoid activation function.

[0106] In a preferred embodiment, to alleviate the long-tail property distribution problem in the dataset, a class-balancing strategy or focal loss (Focal Loss) can be combined.

[0107] d) detection loss : contains the bounding box regression loss.

[0108] The field task loss is calculated as:

[0109] ;

[0110] wherein, , , , are the weight coefficients of the main class contrast loss , the hierarchical loss , the property loss and the detection loss , respectively.

[0111] To cope with the dynamic changes of species and the deployment requirements across sea areas faced by underwater monitoring, the present application adopts a targeted regularization mechanism to suppress feature space drift and maintain cross-domain generalization ability during fine-tuning in underwater in-situ scenes.

[0112] Specifically, by calculating the distance between the visual features extracted by the target model and the targeted auxiliary model for the same input image, a feature space regularization loss is constructed. This loss imposes constraints on the learning direction of the target model, preventing its feature space from deviating too much from a general-purpose visual knowledge space with strong generalization ability when learning domain-specific knowledge.

[0113] Step 5.4, calculate the feature space regularization loss by calculating the distance between the visual features extracted by the target model and the targeted auxiliary model for the same input image. The feature space regularization loss is calculated as follows:

[0114] ;

[0115] wherein represents the visual feature vector extracted by the target model for the i th candidate region, represents the visual feature vector of the i-th candidate region extracted by the target auxiliary model, is an L2 norm symbol.

[0116] Step 5.5, calculate the composite loss based on the domain task loss and the feature space regularization loss as:

[0117] ;

[0118] wherein, is the calculated composite loss, and λ is a weighting coefficient.

[0119] Step 6, unlabeled sample detection and inference: using the optimized target model to detect fish in underwater in-situ images without labels, locating the positions of closed-set fish targets and open-world fish targets and classifying them. Specifically as follows:

[0120] a) Open-set detection mode based on preset categories:

[0121] In this mode, first, an underwater in-situ image to be detected is obtained as input. Second, the target model performs semantic similarity calculation on the visual features of the image with a preset library containing all Base and Novel text prototypes. Finally, this step can output the position and category information of the closed-set fish targets (i.e. labeled training samples) and open-world fish targets (i.e. unlabeled samples) in the image.

[0122] b) Open-vocabulary detection mode based on user instructions:

[0123] In this mode, first, a natural language text instruction input by the user is received, which describes one or more arbitrary target categories. Second, using the text encoder of the target model, the text instruction is encoded into a set of temporary, task-driven text feature vectors in real time. Then, an image to be detected is obtained, and the target model detects the arbitrary target in the user instruction by calculating the similarity between the visual features of the image and the set of temporary text feature vectors. Finally, this step can output the position and category information of the target matching the user instruction in the image.

[0124] Based on the above embodiments, those skilled in the art can easily understand that the present application also provides an underwater in-situ fish unlabeled sample detection device, which comprises a storage medium and a processor, the storage medium stores a computer program, and the computer program is executed by the processor to realize the above-mentioned method.

[0125] Specific case:

[0126] (1) Experimental environment and data set

[0127] a) Experimental environment

[0128] Experimental platform. The experiment was carried out on a server equipped with Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz and 4 NVIDIA A100-PCIE-40GB GPU, running Ubuntu 20.04 operating system. The deep learning framework uses PyTorch 1.12.

[0129] Evaluation index. In target detection, IoU (Intersection over Union, IoU) is used to judge whether the detection box belongs to the detected target. IoU represents the overlap between the detected position information and the real target box . At the same time, IoU is used to predict the accuracy of the position information of the result, and the calculation expression is as follows:

[0130] ;

[0131] Average precision (AP) is a core index to measure the detection performance of a single class. mAP represents the average value of AP of all classes, and the expression is as follows:

[0132] ;

[0133] K represents the number of target classes to be tested, mAP@0.5 (or AP50) represents the mAP value calculated when the IoU threshold is fixed at 0.5. mAP@0.75 (or AP75) represents the mAP value calculated when the IoU threshold is fixed at 0.75. mAP (or AP) is obtained by calculating the AP of each class at 10 different IoU thresholds (from 0.5 to 0.95, step size 0.05), and then averaging all the results.

[0134] b) Data set construction

[0135] The data of the experiment is obtained from the underwater observation platform placed in the sea area of Weizhou Island in Guangxi (east longitude 109°7'32", north latitude 21°4'57") and Yutai Ranch in Weihai, Shandong (east longitude 121°55'50", north latitude 37°28'23"). The in-situ observation video sequence is obtained by the monocular camera of the underwater observation platform, and representative images are extracted from the video sequence. According to the method of the present application, step one is used to construct an image-text structured knowledge base for 28 fish species. According to the method of the present application, step two is used to construct an open vocabulary target detection data set FishDet, and 23 Base classes and 5 Novel classes are obtained.

[0136] (2) Experimental results

[0137] The strict closed set and open set evaluation is performed on the FishDet dataset. Table 1 is the detection result of the target model in the zero-shot setting on the FishDet dataset. In terms of overall performance, the model achieves an AP value of up to 95.9% on 23 closed set categories and 54.9% on 5 open world categories. It is worth noting that the performance of conventional methods on the zero-shot task usually collapses to near zero, which proves that the method of the present application has made substantial progress in overcoming catastrophic forgetting and solving the BNT problem.

[0138] Table 1 is the zero-shot result of the target model on the FishDet dataset

[0139] ;

[0140] As can be seen from the experimental results in Table 2, the method of the present application exhibits extremely excellent and robust performance on the closed set task. Among them, in 8 categories such as black snake, sea cucumber, chest spot snapper, and zebu rudder fish, an AP value of 100% is achieved, and the AP value of individual species such as the mimetic osphronemus is only 50.50%. This may be due to the fact that the body surface of this species is highly similar to the background of the coral reef, and the camouflage pattern makes the visual features not obvious, which is easy to miss or be confused with the background.

[0141] Table 2 is the zero-shot result of the target model on the FishDet dataset Base class

[0142] ;

[0143] As can be seen from the experimental results in Table 3, the method of the present application exhibits excellent generalization ability on the open set zero-shot detection task. The model exhibits excellent zero-shot reasoning ability on multiple Novel categories, achieving an AP value of up to 75.2%, 58.6% and 50.5% on the point-banded spiny scale fish, zebu snapper and gold line blue sub-fish, respectively. This shows that the model learns the unique knowledge of the closed set categories while retaining and utilizing the transferable and general visual semantic concepts (such as “red”, “stripes”, “spindle-shaped body”, etc.) learned from visual-textual semantic space training, thereby identifying completely new species. Secondly, the analysis of the lower performance categories further reveals the challenge of the task and the potential of the method of the present application. For example, the AP value of the brown-spotted long-finned zebu snapper is only 34.76%. This may be due to the fact that the visual features of these new categories are “semantically confused” with one or more closed set categories in the training set, making it difficult for the model to perform accurate zero-shot reasoning.

[0144] Table 3 is the zero-shot result of the target model on the FishDet dataset Novel class

[0145] ;

[0146] In summary, the experimental results on the open set task show that the method can effectively maintain the generalization ability of the model, so that the model can go from the "closed set" to the "open set", providing a feasible technical path for realizing the open world perception in the underwater environment.

[0147] The above only describes the preferred embodiments of the present application, and it should be pointed out that other parts not specifically described belong to the prior art or common knowledge for ordinary skilled in the art. Without departing from the principles of the present application, several improvements and refinements can also be made, which should be considered as the protection scope of the present application.

Claims

1. A method of underwater in situ fish tagless sample detection, characterized in that, The method comprises the following steps: Step 1, obtaining in-situ images captured by a monocular camera of an underwater observation platform, and based on prior knowledge in the field of fisheries, decoupling the fish classes appearing in the images into multi-level classification labels and multiple combinations of visual attribute labels, and constructing an image-text structured knowledge base; Step 2, extracting training and verification data sets of the detection model from the knowledge base, and dividing the data sets into closed set categories and open world categories; Step 3, receiving a natural language text instruction input by a user to describe a to-be-detected data set, and using a text encoder of a visual-language model to encode the natural language text instruction to generate a text feature vector of a target category; Step 4, using a pre-trained visual-language model to perform visual-text semantic space feature alignment training on the image-text pair data set to obtain a targeted auxiliary model; Step 5, constructing a collaborative architecture comprising a target model to be optimized and a regularized reference targeted auxiliary model; dynamically and interactively fusing visual features and category text features of in-situ images; and jointly optimizing the target model in an end-to-end manner based on a structured knowledge guided multi-task learning module and a feature space regularization loss function guided by the targeted auxiliary model; Step 6, using the optimized target model to detect fish appearing in underwater in-situ images without labels, and locating and classifying closed set fish targets and open world fish targets; Step 4 specifically comprises: Step 4.1, initializing a pre-trained visual-language model comprising an image encoder and a text encoder as a base model; Step 4.2, using a data set of closed set categories and a text feature vector of a target category to form an image-text pair for model training; Step 4.3, in the visual-text semantic space training, the weights of the base model are optimized by contrastive learning on the image-text pair, the similarity between the matched image visual features and the text features is maximized, and the similarity between the unmatched image visual features and the text features is minimized, so as to learn the corresponding relationship between the visual concepts and the text descriptions in the field of underwater fish, and the trained model is used as a targeted auxiliary model.

2. The method of claim 1, wherein, The multi-level classification label comprises a main category label and a set of hierarchical classification labels, the main category label is used to uniquely identify the category number and standard name of the fish class, and the hierarchical classification label is used to represent the field of biological hierarchical relationship, comprising the number and name of order and family, and the visual attribute label is used to describe the attribute set of observable morphological characteristics.

3. The method of claim 1, wherein, The knowledge base comprises an image sub-base and a text sub-base, the image sub-base is used for indexing and metadata registration of in-situ images, and the text sub-base is used to provide a structured entry for each category in the fish data set.

4. The method of claim 1, wherein, Step 3 comprises: Category names within the target domain By sequentially embedding one or more predefined static text prompt templates, a set of text sequences is obtained. Each text sequence is input into the text encoder of the vision-language model to obtain a set of corresponding text feature vectors. .

5. The method of claim 1, wherein, Step 5 specifically comprises: Step 5.1, the architecture of the target model uses a visual-language model and a two-stage target detection framework to generate candidate regions using a region proposal network in the target detection framework; Step 5.2, dynamically and interactively fusing the visual features and the textual features of the candidate region of the in-situ image by using a dynamic language perception module; Step 5.3, calculating a domain task loss based on a structured knowledge guided multi-task learning module; Step 5.4, calculating a feature space regularization loss by calculating the distance between the visual features extracted by the target model and the visual features extracted by the targeted auxiliary model for the same input image; Step 5.5, calculating the composite loss based on the domain task loss and the feature space regularization loss as follows: ； wherein, is the computed compound loss, is the domain task loss, is the feature space regularization loss, and λ is a weighting factor.

6. The method of claim 5, wherein the method is performed in situ on a fish without tagging the fish. The step 5.2 specifically comprises: For each category c in the structured knowledge base, a dynamic prompt Prompt c embedded with a learnable continuous vector V is constructed, and all the dynamic prompts corresponding to all categories are input into the text encoder of the target model to generate a set of dynamic text features as text prototypes For the i th candidate region feature A candidate region level cross-attention module is constructed: ； wherein, Query vector as visual features, , , learned weight matrix, , Key and Value vectors as text features, respectively; The attention weights of the candidate region level cross-attention module is represented as: ； wherein, is a normalized exponential function, is a transpose symbol of a matrix, is a key vector dimension; The fusion feature output by the candidate region level cross attention module is represented as: 。 7. The method of claim 6, wherein the method is performed in situ on a fish without tagging the fish. The step 5.3 specifically comprises: taking the candidate region features of the target model or a linear projection thereof, the text prototype of the kth category is , the temperature is , the cosine similarity between the candidate region features of the target model and the text prototype of the kth category is :​ ； wherein, is a similarity calculation function; Adopting hard negative samples, the feature of each candidate region is Selecting the top-K most similar negative class sets The main class contrast loss is calculated : ； wherein, is the scaled cosine similarity between the i-th candidate region and its positive class text prototype, is the total number of candidate regions; The classification results of the genus, species, and family levels are supervised respectively by using a weighted sum of multiple cross-entropy loss terms to obtain a predicted distribution and a real label of each level, and a cross-entropy is used to calculate a level loss as follows: ； wherein, is a hierarchy set, is a weight for different hierarchy, is a cross-entropy loss function representing a criterion; A multi-label binary cross-entropy loss is used to simultaneously predict multiple visual attributes of the target, to obtain a predicted label of an mth attribute of an ith candidate region and a real label , and to calculate an attribute loss as follows: ； wherein, is the total number of attributes, is a sigmoid activation function; Computing field task loss is: ； wherein, to detect loss, , , , are weight coefficients of the main class contrast loss , the hierarchical loss , the attribute loss and the detection loss , respectively.

8. The method of claim 5, wherein, The feature space regularizing loss is calculated as follows: ； wherein, represents a visual feature vector extracted by the target model for the i-th candidate region, represents a visual feature vector extracted by the targeting auxiliary model for the i-th candidate region, is an L2 norm symbol.

9. An underwater in-situ fish tagless sample detection device comprising a storage medium and a processor, the storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-8.

Images