Underwater organism detection method, device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of biological detection, and discloses an underwater organism detection method, device and equipment, a storage medium and a program product.A big language model is used for processing the category name of an underwater organism to be detected to obtain category description, and the name, a picture and the description are input into a target multi-modal detection model together for processing; the dependence on manual annotation is reduced, and the data processing efficiency is improved. Meanwhile, data information of two modes of text and image is fully utilized, the limitation of single-mode data is avoided, and the detection accuracy is improved. Furthermore, by calculating the similarity between the visual features of the target and the false detection clustering center and performing result judgment according to a similarity threshold value, the target possibly causing false detection can be effectively identified, so that the omission ratio can be better balanced, the detection performance reduction caused by improper false detection processing measures is avoided, and the detection accuracy is improved. The problem that the recall rate of the positive sample is reduced is solved, and the reliability of underwater biological detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of biological detection technology, and in particular to an underwater biological detection method, device, equipment, storage medium and program product. Background Art

[0002] At present, underwater biological detection usually uses target detection models such as Faster RCNN and YOLO. These models need to collect and annotate a large number of relevant pictures before training. Underwater biological pictures can be obtained by diving or underwater drone photography, and image annotation needs to be done manually. When the amount of data is large, the entire collection and annotation process will be lengthy, time-consuming and labor-intensive.

[0003] At the same time, this type of model does not have a good mechanism for eliminating false positives. When a false positive occurs during the detection process, the existing approach is to reduce false positives by increasing the classification confidence threshold or continuing to collect data to fine-tune the model. In the former, false positives will be reduced as the threshold is increased, but the recall rate of positive samples will also be reduced, and then missed detections will increase; the latter requires continued data collection, labeling, and retraining of the model, but the retrained model may not necessarily eliminate the current false positives, and may even generate new false positives. Therefore, neither of these two solutions can effectively eliminate false positives. Summary of the invention

[0004] In view of this, the present invention provides an underwater biological detection method, device, equipment, storage medium and program product to solve the problem that the collection and labeling process in the existing underwater biological detection process is lengthy, time-consuming and labor-intensive, and cannot effectively eliminate false detections.

[0005] In a first aspect, the present invention provides a method for detecting underwater organisms, the method comprising:

[0006] The name of the underwater creature category to be detected, the image of the underwater creature category to be detected and the false detection cluster centers of multiple underwater creature categories are obtained; the name of the underwater creature category to be detected is input into the large language model for processing to obtain the description of the underwater creature category to be detected; the name of the underwater creature category to be detected, the image of the underwater creature category to be detected and the description of the underwater creature category to be detected are input into the target multimodal detection model for processing to obtain multiple target visual features and multiple underwater creature initial detection results; the similarity between each target visual feature and each underwater creature category false detection cluster center is calculated to obtain multiple similarity values; the multiple similarity values ​​are respectively compared with the preset similarity thresholds, and multiple underwater creature target detection results are determined according to the comparison results and the multiple underwater creature initial detection results.

[0007] The underwater organism detection method provided by the present invention uses a large language model to process the category name of the underwater organism to be detected to obtain a category description, and inputs the name, picture and description into the target multimodal detection model for processing, thereby reducing the reliance on manual annotation and improving the efficiency of data processing, especially when processing a large amount of data, avoiding a lengthy data annotation process. At the same time, the data information of the two modes of text and image is fully utilized, and then the multimodal fusion helps the model to understand the characteristics of underwater organisms more comprehensively, avoiding the limitations of single modal data, thereby improving the accuracy of detection. Further, by calculating the similarity between the target visual features and the false detection cluster center, and judging the results according to the similarity threshold, the target that may cause false detection can be effectively identified, and then the target similar to the false detection feature can be excluded in advance during the detection process, while reducing false detection, the missed detection rate can be better balanced, avoiding the decline in detection performance caused by improper false detection processing measures, and solving the problem that the traditional method often leads to a decrease in the positive sample recall rate when reducing the false detection rate, thereby increasing the missed detection rate. Therefore, by implementing the present invention, the reliability of underwater organism detection is improved.

[0008] In an optional implementation, obtaining multiple underwater organism category misdetection cluster centers includes:

[0009] Acquire multiple historical false detection targets; input the multiple historical false detection targets into the visual encoder of the target multimodal detection model to obtain multiple historical false detection target features; perform cluster analysis on the multiple historical false detection target features to obtain multiple underwater organism category false detection cluster centers.

[0010] The underwater organism detection method provided by the present invention fully utilizes the data in the previous detection process by acquiring multiple historical false detection targets. Furthermore, the multiple historical false detection targets are input into the visual encoder of the target multimodal detection model for processing, and the visual encoder can extract the key visual features of these false detection targets. Finally, cluster analysis is performed on the features of multiple historical false detection targets to discover the similarities and regularities between the false detection targets, so that the obtained multiple underwater organism category false detection cluster centers can well represent different types of false detection situations, providing a basis for judging whether it is a false detection target in subsequent detection.

[0011] In an optional implementation, obtaining multiple historical false positive targets includes:

[0012] Acquire multiple first underwater creature category names and multiple first underwater creature category pictures; process the multiple first underwater creature category names and multiple first underwater creature category pictures through a large language model and a target multimodal detection model to obtain multiple underwater creature detection results; and determine multiple historical false detection targets based on the multiple underwater creature detection results.

[0013] The underwater organism detection method provided by the present invention can reflect the false detection of the model in the real detection scene by obtaining the name and picture of the first underwater organism category and determining the historical false detection target through model processing, and provides an accurate data basis for subsequent false detection analysis and model optimization. Furthermore, determining the false detection target based on the actual detection results is conducive to continuous iteration and improvement of the underwater organism detection process, and then by analyzing the causes of these false detection targets in the future, the model and detection process can be optimized in a targeted manner, and the performance of the entire detection process can be improved.

[0014] In an optional embodiment, the method further includes:

[0015] Acquire a plurality of second underwater creature category names and a preset number of second underwater creature category pictures;

[0016] A plurality of second underwater creature category names and a preset number of second underwater creature category pictures are processed by a large language model and a preset target multimodal detection model to obtain a target box classification loss and a target box position regression loss; the preset multimodal detection model is adjusted using the target box classification loss and the target box position regression loss to obtain a target multimodal detection model.

[0017] The underwater biological detection method provided by the present invention uses the existing preset multimodal detection model as a basis, avoids the complex process of building a model from scratch, and saves a lot of computing resources and time costs. Further, based on the name of the second underwater biological category and a preset number of underwater biological category pictures, after being processed by the large language model and acting together with the preset multimodal detection model, the target frame classification loss and the target frame position regression loss are obtained, which can more accurately measure the error of the model in the underwater biological detection task. Further, the preset multimodal detection model is adjusted according to the calculated target frame classification loss and target frame position regression loss, so that the model can quickly focus on the key issues in the underwater biological detection task, reducing unnecessary training time and data volume, and improving the pertinence and effectiveness of model training. Therefore, by implementing the present invention, using the pre-trained multimodal model, only a small amount of pictures need to be collected to fine-tune the model to achieve subsequent underwater biological detection, saving labor costs.

[0018] In an optional embodiment, the target multimodal detection model includes a text encoder, a visual encoder, a graphic feature fusion module and a detection head;

[0019] The name of the underwater organism category to be detected, the picture of the underwater organism category to be detected, and the description of the underwater organism category to be detected are input into the target multimodal detection model for processing, and multiple target visual features and multiple underwater organism initial detection results are obtained, including:

[0020] The names of the underwater creature categories to be detected and the descriptions of the underwater creature categories to be detected are processed by a text encoder to obtain multiple text features; the pictures of the underwater creature categories to be detected are processed by a visual encoder to obtain multiple initial visual features; the multiple text features and the multiple initial visual features are processed by a graphic feature fusion module and a detection head respectively to obtain multiple target classification confidences and multiple target detection frame positions; multiple underwater creature initial detection results are generated based on the multiple target classification confidences and the multiple target detection frame positions; the multiple initial visual features and the multiple target detection frame positions are processed by a preset regional feature aggregation algorithm to obtain multiple target visual features.

[0021] The underwater life detection method provided by the present invention utilizes the text encoder in the target multimodal detection model to process the name of the underwater life category to be detected and the description of the underwater life category to be detected, and can accurately convert the semantic information in the text into multiple text features, so that the subsequent model can deeply understand the key characteristics contained in different underwater life categories at the text level. At the same time, the visual encoder processes the picture of the underwater life category to be detected, and the actual visual information of the underwater life in the picture can be captured. Further, the multiple text features and the multiple initial visual features are processed by the graphic feature fusion module respectively, so that the text information and the image information can be deeply integrated at the feature level, and the overall feature expression of the underwater life can be enhanced. Further, through further calculation by the detection head, multiple target classification confidences and multiple target detection frame positions can be obtained and multiple underwater life initial detection results can be generated, which makes full use of the obtained classification possibility and position information, and avoids the misjudgment that may be caused by judging based on a single parameter. Finally, multiple initial visual features and multiple target detection box positions are processed by a preset regional feature aggregation algorithm, which can more accurately extract the features of the corresponding target area from the image features, solving the problems of inaccurate regional positioning, feature loss or confusion that may exist in traditional feature extraction methods.

[0022] In an optional implementation, the plurality of similarity values ​​are respectively compared with a preset similarity threshold, and a plurality of underwater organism target detection results are determined according to the comparison results and a plurality of underwater organism initial detection results, including:

[0023] The multiple similarity values ​​are respectively compared with the preset similarity threshold value; according to the comparison results, the underwater organism initial detection results that do not meet the conditions are filtered out from the multiple underwater organism initial detection results to obtain multiple underwater organism target detection results, wherein the underwater organism initial detection results that do not meet the conditions represent the detection results of underwater organisms whose similarity values ​​are greater than the preset similarity threshold value.

[0024] The underwater organism detection method provided by the present invention compares the similarity value between the target visual features and the false detection cluster center with a preset similarity threshold, and can accurately identify targets with similar false detection features and filter them out from the initial detection results, effectively improving the accuracy of the final detection results. Furthermore, the final underwater organism target detection result is determined by combining the target visual features, target classification confidence, and target detection frame position, and by comparing the similarity with the false detection cluster center, thereby achieving accurate detection of target underwater organisms in complex underwater environments.

[0025] In a second aspect, the present invention provides an underwater biological detection device, the device comprising:

[0026] The first acquisition module is used to obtain the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and multiple false detection cluster centers of the underwater creature category; the first processing module is used to input the name of the underwater creature category to be detected into the large language model for processing, and obtain the description of the underwater creature category to be detected; the second processing module is used to input the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected into the target multimodal detection model for processing, and obtain multiple target visual features and multiple underwater creature initial detection results; the calculation module is used to calculate the similarity between each target visual feature and each underwater creature category false detection cluster center to obtain multiple similarity values; the comparison and determination module is used to compare the multiple similarity values ​​with the preset similarity threshold respectively, and determine multiple underwater creature target detection results according to the comparison results and the multiple underwater creature initial detection results.

[0027] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the underwater biological detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the underwater biological detection method of the first aspect or any corresponding embodiment thereof.

[0029] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the underwater biological detection method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a schematic flow chart of an underwater organism detection method according to an embodiment of the present invention;

[0032] Figure 2 is a schematic flow chart of another underwater organism detection method according to an embodiment of the present invention;

[0033] Figure 3 is a schematic flow chart of another underwater organism detection method according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of a model training process according to an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of a false detection target clustering process according to an embodiment of the present invention;

[0036] Figure 6 is a schematic diagram of a model reasoning process according to an embodiment of the present invention;

[0037] Figure 7 is a structural block diagram of an underwater biological detection device according to an embodiment of the present invention;

[0038] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0040] An embodiment of the present invention provides an underwater organism detection method, which achieves the effect of improving the reliability of underwater organism detection by combining multiple underwater organism category false detection cluster centers, a large language model, a target multimodal detection model, etc.

[0041] According to an embodiment of the present invention, an embodiment of an underwater biological detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] In this embodiment, a method for detecting underwater organisms is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of an underwater organism detection method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0043] Step S101, obtaining the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected, and multiple underwater creature category false detection cluster centers.

[0044] The number of the names of the underwater creature categories to be detected and the number of the pictures of the underwater creature categories to be detected can be one or more, which is not specifically limited in this embodiment.

[0045] Furthermore, the names of the categories of underwater organisms to be detected are stored in a text format.

[0046] Specifically, the scope of underwater organisms to be detected can be determined based on the specific needs and application scenarios of underwater organism detection. For example, if the ecological monitoring is for a specific sea area, the underwater organism categories such as "clownfish", "seahorse", "coral reef" and "sea turtle" may be determined as the underwater organism categories to be detected based on the existing underwater organism census data or research focus of the sea area.

[0047] Furthermore, the names of these determined underwater biological categories can be recorded one by one and saved in text format.

[0048] Furthermore, for each type of target, a crawler tool can be used to collect relevant images on the Internet, or an underwater camera can be carried in the ocean to collect biological images to collect corresponding images, i.e., images of the underwater biological category to be detected.

[0049] Furthermore, for the collected images, a rectangular frame can be used to mark the location of the target organism in the image and different labels can be assigned.

[0050] Furthermore, the cluster centers of false positive detections of multiple underwater organism categories can reflect the characteristic patterns that are prone to false positive detections during the detection process.

[0051] Step S102, input the name of the underwater creature category to be detected into the large language model for processing to obtain a description of the underwater creature category to be detected.

[0052] The large language model may be any large language model based on deep learning, including but not limited to Llama, Qwen, ChatGPT, etc.

[0053] Specifically, the name of the underwater organism category to be detected is input into the large language model for processing, and a detailed description of the category can be obtained.

[0054] Step S103, inputting the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected into the target multimodal detection model for processing, and obtaining multiple target visual features and multiple underwater creature initial detection results.

[0055] Specifically, after inputting the name of the underwater organism category to be detected, the picture of the underwater organism category to be detected and the description of the underwater organism category to be detected into the target multimodal detection model, the underwater organisms are detected through the target multimodal detection model to output corresponding multiple target visual features and multiple underwater organism initial detection results.

[0056] Furthermore, in an optional implementation, the number of the target visual feature may also be one.

[0057] Step S104, calculating the similarity between each target visual feature and each underwater organism category misdetected cluster center to obtain a plurality of similarity values.

[0058] Specifically, the similarity value between each target visual feature and the false detection cluster center of each underwater organism category can be calculated by a similarity measurement method.

[0059] Among them, the similarity measurement method can be Euclidean distance, cosine similarity, Manhattan distance and other methods.

[0060] Step S105 , comparing the multiple similarity values ​​with the preset similarity thresholds respectively, and determining multiple underwater organism target detection results according to the comparison results and the multiple underwater organism initial detection results.

[0061] Specifically, each obtained similarity value is compared with a preset similarity threshold.

[0062] Furthermore, the similarity between each target visual feature and each underwater organism category false detection cluster center can be determined through the comparison results, and then the final detection result of the corresponding underwater organism can be further determined in combination with the initial underwater organism detection result.

[0063] The underwater organism detection method provided in this embodiment uses a large language model to process the name of the underwater organism category to be detected to obtain a category description, and inputs the name, picture and description into the target multimodal detection model for processing, thereby reducing the reliance on manual annotation and improving the efficiency of data processing, especially when processing a large amount of data, avoiding the lengthy data annotation process. At the same time, the data information of the two modes of text and image is fully utilized, and then the multimodal fusion helps the model to understand the characteristics of underwater organisms more comprehensively, avoiding the limitations of single modal data, thereby improving the accuracy of detection. Further, by calculating the similarity between the target visual features and the false detection cluster center, and judging the results according to the similarity threshold, the target that may cause false detection can be effectively identified, and then the target similar to the false detection feature can be excluded in advance during the detection process. While reducing false detection, the missed detection rate can be better balanced, avoiding the decline in detection performance caused by improper false detection processing measures, and solving the problem that the traditional method often leads to a decrease in the recall rate of positive samples when reducing the false detection rate, thereby increasing the missed detection rate. Therefore, by implementing the present invention, the reliability of underwater organism detection is improved.

[0064] In this embodiment, a method for detecting underwater organisms is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 2 is a flow chart of an underwater organism detection method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0065] Step S201, obtaining the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected, and multiple underwater creature category false detection cluster centers.

[0066] Among them, the acquisition of the name of the underwater creature category to be detected and the picture of the underwater creature category to be detected can refer to the description of the above step S101, which will not be repeated here.

[0067] Furthermore, the cluster centers of multiple underwater organism category misdetections can be obtained by the following steps:

[0068] Step a1, obtaining multiple historical false detection targets.

[0069] Among them, historical false detection targets refer to objects that were misjudged by the model in the past when the relevant detection model was used to detect underwater organisms.

[0070] In an optional implementation, the above step a1 includes:

[0071] Step a11, obtaining a plurality of first underwater creature category names and a plurality of first underwater creature category pictures.

[0072] Step a12, processing the multiple first underwater life category names and the multiple first underwater life category pictures through the large language model and the target multimodal detection model to obtain multiple underwater life detection results.

[0073] Step a13, determining multiple historical false detection targets based on multiple underwater organism detection results.

[0074] Specifically, the corresponding multiple first underwater creature category pictures can be obtained by the following acquisition method:

[0075] (1) Collecting images through web crawlers: First, we need to select websites that are legal and compliant and contain rich underwater biological image resources as data sources. Secondly, we need to choose a suitable web crawler framework, such as the Scrapy framework in Python or BeautifulSoup combined with the requests library to build a crawler.

[0076] Furthermore, according to the requirements of the selected framework, the corresponding crawler code is written, and key parameters are configured, such as the target website URL to be crawled, the image storage path, the search keywords (i.e. the names of the various underwater creature categories determined earlier), etc. Finally, the written crawler program is run, and the underwater creature images that meet the requirements are searched on the web page according to the set keywords.

[0077] (2) Use underwater cameras to collect pictures on site: arrange professional marine scientific research personnel to carry appropriate underwater shooting equipment (such as underwater cameras, underwater video cameras, etc.) to dive into the target sea area for shooting. During the shooting process, try to cover different marine environment areas (such as coral reef areas in shallow waters, near submarine mountains in deep seas, etc.) to ensure that the underwater organisms to be tested in various states can be photographed. At the same time, the shooting angles should be diversified to capture the morphological characteristics of underwater organisms in all directions, such as photographing fish from different sides, photographing coral reefs from above and from the side, etc.

[0078] Furthermore, the acquired multiple first underwater life category names may be input into a large language model for processing and a category description of each underwater life category may be output.

[0079] Furthermore, a plurality of first underwater life category names, a plurality of category descriptions of the underwater life categories, and a plurality of first underwater life category pictures are input into the target multimodal detection model.

[0080] Furthermore, the target multimodal detection model can extract the visual features of the image through its internal visual encoder, and at the same time combine components such as the text encoder, the image-text feature fusion module and the detection head to perform underwater organism detection and output corresponding multiple underwater organism detection results.

[0081] Furthermore, multiple historical false detection targets can be determined from multiple underwater biological detection results based on human experience.

[0082] Step a2, inputting multiple historical false detection targets into the visual encoder of the target multimodal detection model to obtain multiple historical false detection target features.

[0083] Specifically, the original images corresponding to the multiple falsely detected targets may be input into the target multimodal detection model, and the visual encoder of the target multimodal detection model may be used to perform feature extraction and obtain the corresponding multiple falsely detected target features.

[0084] Step a3: cluster analysis is performed on multiple historical false positive target features to obtain multiple false positive cluster centers of underwater organism categories.

[0085] The preset clustering method may be a Kmeans clustering algorithm.

[0086] Specifically, a preset clustering method may be used to perform clustering processing on multiple false detection target features, and obtain corresponding multiple false detection cluster centers of underwater organism categories.

[0087] In an optional implementation, taking the Kmeans clustering algorithm as an example, the number of clusters can be determined first, such as by visually analyzing the distribution of multiple falsely detected target features (for example, using dimensionality reduction techniques such as t-SNE to map high-dimensional feature vectors to a two-dimensional plane to observe their distribution trends and roughly determine the number of clusters that may be appropriate), or based on the experience of similar projects in the past, conducting multiple experiments with different numbers of clusters and comparing the clustering effects (for example, observing whether the intra-class distance is small enough, whether the inter-class distance is large enough, and other indicators to evaluate the rationality of clustering) to determine a relatively appropriate number of clusters.

[0088] Furthermore, the Kmeans algorithm can group all falsely detected target feature vectors according to the similarity between the falsely detected target features (usually using Euclidean distance as the similarity metric to calculate the distance between each feature vector, the closer the distance, the higher the similarity) and divide them into a set number of different clusters.

[0089] Furthermore, during the iteration of the Kmeans algorithm, the center position of each cluster can be continuously adjusted (the cluster center is actually an average representation of all feature vectors in the cluster in the feature space, and its value will be continuously updated and optimized with the iteration of the algorithm, so that the sum of the distances from the feature vectors in the cluster to the cluster center is as small as possible) until the preset convergence conditions are met (such as reaching the set number of iterations or the position change of the cluster center is less than a certain threshold, etc.), at which point the clustering process ends.

[0090] Finally, after calculation and iteration of the Kmeans algorithm, multiple cluster centers (consistent with the number of the set n_clusters parameter) can be obtained. Furthermore, each cluster center is a typical representative of the false detection target feature in the corresponding cluster, and embodies the common feature pattern of the features in the cluster in the form of target features.

[0091] Step S202: Input the name of the underwater creature category to be detected into the large language model for processing to obtain the description of the underwater creature category to be detected. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0092] Step S203, obtaining a plurality of second underwater life category names and a preset number of second underwater life category pictures.

[0093] Among them, the preset number of second underwater life category pictures is a small number of underwater life category pictures. For example, under normal circumstances, thousands or tens of thousands of underwater life category pictures need to be collected, but in this embodiment, only hundreds or dozens of underwater life category pictures need to be collected.

[0094] Furthermore, the locations of target organisms in the collected images are marked with rectangular frames and given different labels.

[0095] The specific acquisition process can refer to the relevant description in the above step S101, which will not be repeated here.

[0096] Step S204, processing the plurality of second underwater life category names and a preset number of second underwater life category pictures through a large language model and a preset target multimodal detection model to obtain a target frame classification loss and a target frame position regression loss.

[0097] Among them, the preset target multimodal detection model can be any multimodal pre-trained detection model based on deep learning, including but not limited to GLIP, GroundingDino, YoloWorld and other models.

[0098] Specifically, the pre-training parameters of the multimodal detection model are preset, wherein the training data includes a triplet of <image, category name, category description>.

[0099] Furthermore, the names of multiple second underwater organism categories are processed by the large language model and output to obtain a detailed description of each category.

[0100] Furthermore, multiple second underwater life category names, a preset number of second underwater life category pictures and the obtained detailed description of each category are input into a preset multimodal detection model for processing, and the target frame classification loss and the target frame position regression loss can be obtained.

[0101] Among them, the target box classification loss can be used to measure the degree of difference between the underwater organism category predicted by the model and the actual category; the target box position regression loss can be used to measure the deviation between the position of the underwater organism in the picture predicted by the model (represented by the bounding box coordinates) and the actual position.

[0102] Specifically, after multiple second underwater biological category names, a preset number of second underwater biological category pictures and the obtained detailed description of each category are input into the preset multimodal detection model, they are processed by the text encoder, visual encoder, and image-text feature fusion module of the preset multimodal detection model to obtain corresponding multiple image-text fusion features.

[0103] Furthermore, the image-text fusion features are input into the detection head of the preset multimodal detection model, and the corresponding target box classification loss and target box position regression loss can be calculated and output.

[0104] In an optional embodiment, the detection head of the target multimodal detection model includes a classification submodule and a regression submodule.

[0105] Specifically, after the image-text fusion features are input into the classification submodule, the classification submodule can analyze the image-text fusion features, determine the category of the corresponding underwater organism, and then compare the predicted category with the actual category (annotated in the second underwater organism category picture, that is, the known accurate category).

[0106] Furthermore, a suitable loss function (such as the cross entropy loss function, etc.) can be used to measure the difference between the predicted category and the actual category, and the calculated loss value is the target box classification loss.

[0107] Furthermore, when the image-text fusion features are input into the regression submodule, the regression submodule can determine the position of the underwater creatures in the picture based on the image-text fusion feature vector, output the predicted bounding box coordinates (i.e., the target detection box position), and then compare it with the actual marked accurate position of the underwater creatures in the picture (which also has corresponding annotations in the training data).

[0108] Furthermore, a suitable loss function (such as mean square error loss function, etc.) can be used to measure the deviation between the predicted position and the actual position, and the obtained loss value is the target box position regression loss.

[0109] Step S205: The preset multimodal detection model is adjusted using the target box classification loss and the target box position regression loss to obtain a target multimodal detection model.

[0110] Specifically, after obtaining the target box classification loss and the target box position regression loss, the back propagation algorithm can be used to update the parameters of the preset multimodal detection model and construct the final target multimodal detection model.

[0111] Among them, the backpropagation algorithm represents a commonly used supervised learning algorithm for training artificial neural networks.

[0112] In an optional implementation, the model can be adjusted by the following steps:

[0113] (1) Setting training parameters, which may include determining the learning rate, setting the number of training rounds, selecting the optimizer, and other parameters (such as batch size, etc.).

[0114] (2) Using the back-propagation algorithm, based on the calculated target box classification loss and target box position regression loss, according to the set learning rate, number of training rounds, optimizer (if selected), batch size and other parameters, starting from the detection head, the loss value is back-propagated to each level of the preset multimodal detection model, and the various parameters of the model (including the parameters of the image encoder, text encoder, image-text feature fusion module, etc.) are gradually adjusted.

[0115] In each round of training, the back propagation algorithm can calculate the gradient information corresponding to each parameter based on the loss value, and then update the value of each parameter according to the parameter settings such as the learning rate, so that the model is optimized in the direction of reducing the loss value and improving the detection accuracy.

[0116] (3) Iterative training and convergence judgment: The model undergoes multiple rounds of iterative training. During the training process, the changes in the target box classification loss and the target box position regression loss are closely monitored. Usually, the convergence of the model can be judged by observing the change curve of the loss value as the number of training rounds increases. For example, if the loss value gradually decreases and tends to stabilize with the increase in the number of training rounds (for example, the loss value drops to a very low level and no longer changes significantly), or a pre-set convergence condition is reached (such as the loss value is less than a certain threshold), then it can be considered that the model training is basically completed and the corresponding target multimodal detection model is obtained.

[0117] Step S206: Input the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected, and the description of the underwater creature category to be detected into the target multimodal detection model for processing, and obtain multiple target visual features and multiple underwater creature initial detection results. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0118] Step S207, calculate the similarity between each target visual feature and each underwater organism category false detection cluster center, and obtain multiple similarity values. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0119] Step S208, compare the multiple similarity values ​​with the preset similarity thresholds respectively, and determine multiple underwater organism target detection results based on the comparison results and the multiple underwater organism initial detection results. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0120] The underwater life detection method provided in this embodiment makes full use of the data in the previous detection process by acquiring multiple historical false detection targets. Further, the multiple historical false detection targets are input into the visual encoder of the target multimodal detection model for processing, and the visual encoder can extract the key visual features of these false detection targets. Finally, cluster analysis is performed on the features of multiple historical false detection targets, and the similarities and regularities between the false detection targets can be found, so that the obtained multiple underwater life category false detection cluster centers can well represent different types of false detection situations. Further, using the existing preset multimodal detection model as a basis, the complex process of building a model from scratch is avoided, saving a lot of computing resources and time costs. Further, based on the second underwater life category name and a preset number of underwater life category pictures, after being processed by the large language model and acting together with the preset multimodal detection model, the target frame classification loss and the target frame position regression loss are obtained, which can more accurately measure the error of the model in the underwater life detection task. Furthermore, the preset multimodal detection model is adjusted according to the calculated target frame classification loss and target frame position regression loss, so that the model can quickly focus on the key issues in the underwater biological detection task, reducing unnecessary training time and data volume, and improving the pertinence and effectiveness of model training. Therefore, by implementing the present invention, using the pre-trained multimodal model, only a small amount of pictures need to be collected to fine-tune the model to achieve subsequent underwater biological detection, saving labor costs.

[0121] In this embodiment, a method for detecting underwater organisms is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 3 is a flow chart of an underwater organism detection method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0122] Step S301, obtain the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and multiple underwater creature category false detection cluster centers. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0123] Step S302: Input the name of the underwater creature category to be detected into the large language model for processing to obtain the description of the underwater creature category to be detected. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0124] Step S303, input the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected into the target multimodal detection model for processing, and obtain multiple target visual features and multiple underwater creature initial detection results.

[0125] According to the description of step a13, the target multimodal detection model includes a text encoder, a visual encoder, a graphic feature fusion module and a detection head.

[0126] Specifically, the above step S303 includes:

[0127] Step S3031, the name of the underwater creature category to be detected and the description of the underwater creature category to be detected are processed by a text encoder to obtain multiple text features.

[0128] Specifically, the category name of the underwater organism to be detected and the category description of the underwater organism to be detected can be input into the text encoder of the target multimodal detection model, and the category name and category description can be encoded using the text encoder to obtain category name features and category description features, and then the category name features and category description features can be concatenated to obtain category text features.

[0129] Among them, the text encoder can map the semantic information in the category name and category description into the vector space based on its internal word embedding and other technologies, and obtain the category name feature vector and the category description feature vector respectively. Then, these two feature vectors are spliced ​​together (for example, directly spliced ​​together in dimensional order) to obtain the final category text feature vector, so that the text features can integrate the semantic information of the category name and detailed description, and more comprehensively reflect the characteristics related to the underwater biological category.

[0130] Step S3032: Process the image of the underwater creature category to be detected through a visual encoder to obtain multiple initial visual features.

[0131] Specifically, a picture of the underwater organism category to be detected can be input into the visual encoder of the target multimodal detection model for image encoding to obtain a plurality of initial visual features.

[0132] Among them, according to the algorithm logic set inside the target multimodal detection model, the image encoder can convert the picture into a corresponding image feature vector. Furthermore, the image feature vector can capture the key visual features of the underwater organisms in the picture, such as the shape, texture, color distribution of the underwater organisms, and the feature representation corresponding to the underwater environment background information.

[0133] Step S3033, the multiple text features and the multiple initial visual features are processed by a text-image feature fusion module to obtain multiple text-image fusion features.

[0134] Specifically, the image-text feature fusion module usually uses an attention mechanism-based method or other feature interaction algorithms to achieve fusion.

[0135] In an optional implementation, taking the attention mechanism as an example, the image-text feature fusion module can calculate the correlation weight between the text features and the visual features. For each text feature vector, its correlation with each initial visual feature vector can be evaluated, and vice versa.

[0136] Furthermore, the text features and visual features can be weightedly fused according to the calculated correlation weights. For each text feature vector, the corresponding visual feature vectors can be weighted summed according to the correlation weights with each visual feature vector to obtain a plurality of fused text-related features. Similarly, for each visual feature vector, the corresponding text feature vectors can be weighted summed according to the correlation weights with each text feature vector to obtain a plurality of fused visual-related features.

[0137] Furthermore, the fused multiple text-related features and the fused multiple visual-related features are further combined (such as splicing or other combination methods) to obtain the final multiple image-text fusion features.

[0138] Through fusion, image features and text features can be correlated and complement each other, so that the fused features can not only reflect the visual content in the picture, but also integrate the semantic information of the text, and better serve the subsequent target detection tasks.

[0139] Step S3034, multiple image-text fusion features are processed by a detection head to obtain multiple target classification confidences and multiple target detection frame positions.

[0140] Specifically, multiple image-text fusion features can be input into the detection head of the target multimodal detection model to obtain multiple target classification confidences and multiple target detection box positions.

[0141] In an optional embodiment, the detection head of the target multimodal detection model includes a classification submodule and a regression submodule.

[0142] Specifically, when the image-text fusion features are input into the classification submodule, the neural network layer inside the classification submodule (usually built based on structures such as the fully connected layer) will further extract and analyze the input features, and mine the key information related to different underwater biological categories in the image-text fusion features. For example, if the fusion features contain information related to the morphological characteristics (from the image perspective) and habit descriptions (from the text perspective) of a certain fish, the classification submodule will try to capture these clues to determine whether it belongs to that fish category.

[0143] Furthermore, different neural network layers will gradually transform and abstract the features and map them into a suitable semantic space for category judgment.

[0144] Furthermore, after being processed by a multi-layer neural network, the classification submodule can eventually convert the features into probability values ​​corresponding to each underwater biological category, that is, the target classification confidence, through an activation function (such as the Softmax activation function).

[0145] Furthermore, when the image-text fusion features are input into the regression submodule, the regression submodule can utilize its internal neural network layer (e.g., including convolutional layers, fully connected layers, etc., by learning the mapping relationship between the biological position and features in the image) to extract key information that can indicate the biological position from the image-text fusion features.

[0146] For example, based on the visual features such as the relative position and size of the organisms in the image and the descriptive information such as the distribution environment of the organisms in the text, the regression submodule can convert this comprehensive information into specific coordinate values ​​and try to accurately locate the position of the organisms in the image.

[0147] Furthermore, through a series of calculations and mappings, the regression submodule can finally output the coordinate information of the target detection box position, usually in the form of a rectangular box, such as outputting the coordinates of the upper left corner as (x1, y1) and the coordinates of the lower right corner as (x2, y2), which can clearly define the area where the model believes the underwater creatures are located in the image.

[0148] Furthermore, after parallel processing by the classification submodule and regression submodule of the detection head, the corresponding target classification confidence and target detection box position can be obtained simultaneously for each input image-text fusion feature.

[0149] Step S3035, generating multiple underwater life initial detection results based on multiple target classification confidences and multiple target detection frame positions.

[0150] Specifically, the target category can be determined according to the target classification confidence.

[0151] In an optional implementation, a confidence threshold may be set, and if the confidence corresponding to a certain category exceeds the threshold, the target is preliminarily determined to be of that category. For example, if the confidence threshold is 0.5, for the confidence vector [0.1, 0.8, 0.1] obtained above, the target will be determined to be a hippocampus category.

[0152] Furthermore, the corresponding target detection frame position is associated with the determined category and a corresponding initial detection result is formed.

[0153] Furthermore, the above operation is repeated for all targets to be detected and multiple initial underwater organism detection results are generated.

[0154] Step S3036, processing the multiple initial visual features and the multiple target detection frame positions through a preset regional feature aggregation algorithm to obtain multiple target visual features.

[0155] The preset regional feature aggregation algorithm can be RoiAlign (Region of Interest Align) algorithm refers to an algorithm used to extract and align features of the region of interest (ROI) in computer vision tasks such as target detection and image segmentation.

[0156] Specifically, the obtained multiple initial visual features and multiple target detection box positions are input into the RoiAlign algorithm for processing and obtaining corresponding multiple target visual features, which may include the following steps:

[0157] (1) Determine the region of interest (ROI): The corresponding region of interest can be determined on the initial visual features according to the input target detection box position. Since the target detection box is the coordinate defined on the original image, it needs to be mapped to the coordinate space corresponding to the feature map.

[0158] (2) Feature extraction in a determined region of interest: For each sampling point in the region of interest, the feature value can be obtained from the surrounding integer coordinate points through bilinear interpolation according to its non-integer coordinate position on the feature map.

[0159] (3) Generate multiple target visual features: The above feature extraction operation is performed on the region of interest corresponding to each target detection frame, and finally multiple target visual features are obtained, which can better represent the characteristics of underwater organisms in the target detection frame.

[0160] Step S304, calculate the similarity between each target visual feature and each underwater creature category false detection cluster center, and obtain multiple similarity values. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0161] Step S305 , comparing the multiple similarity values ​​with the preset similarity thresholds respectively, and determining multiple underwater organism target detection results according to the comparison results and the multiple underwater organism initial detection results.

[0162] Specifically, the above step S305 includes:

[0163] Step S3051, comparing the multiple similarity values ​​with the preset similarity thresholds respectively.

[0164] Specifically, each similarity value may be compared with a preset similarity threshold in turn.

[0165] Step S3052, based on the comparison result, filtering the underwater life initial detection results that do not meet the conditions from the multiple underwater life initial detection results to obtain multiple underwater life target detection results.

[0166] The initial underwater life detection results that do not meet the conditions represent the detection results of underwater life with similarity values ​​greater than a preset similarity threshold.

[0167] Specifically, if the similarity value between a certain target visual feature and the false detection cluster center of the underwater organism category exceeds a preset similarity threshold, it means that the target visual feature is likely to correspond to a false detection target, and it is filtered out.

[0168] Furthermore, only when the similarity value is lower than a preset similarity threshold value, the corresponding target visual feature can be determined as an accurate target visual feature.

[0169] Furthermore, after determining the target visual features whose similarity values ​​are lower than the preset similarity threshold, the underwater organism detection results corresponding to the target visual features whose similarity values ​​are greater than the preset similarity threshold can be filtered out from the multiple underwater organism initial detection results, and the final accurate and reliable multiple underwater organism target detection results can be obtained.

[0170] The underwater life detection method provided in this embodiment uses the text encoder in the target multimodal detection model to process the name of the underwater life category to be detected and the description of the underwater life category to be detected, which can accurately convert the semantic information in the text into multiple text features, so that the subsequent model can deeply understand the key characteristics of different underwater life categories at the text level. At the same time, the visual encoder processes the picture of the underwater life category to be detected, and the actual visual information of the underwater life in the picture can be captured. Further, the multiple text features and the multiple initial visual features are processed by the graphic feature fusion module respectively, which can make the text information and the image information deeply integrated at the feature level, and enhance the overall feature expression of the underwater life. Further, through further calculation by the detection head, multiple target classification confidences and multiple target detection frame positions can be obtained and multiple underwater life initial detection results can be generated, which makes full use of the obtained classification possibility and position information, and avoids the misjudgment that may be caused by judging based on a single parameter. Finally, multiple initial visual features and multiple target detection frame positions are processed by a preset regional feature aggregation algorithm, which can more accurately extract the features of the corresponding target area from the image features, solving the problems of inaccurate regional positioning, feature loss or confusion that may exist in traditional feature extraction methods. Furthermore, by comparing the similarity value between the target visual features and the false detection cluster center with the preset similarity threshold, targets similar to the false detection features can be accurately identified and filtered out from the initial detection results, effectively improving the accuracy of the final detection results. Furthermore, by combining the target visual features, target classification confidence and target detection frame position, and comparing the similarity with the false detection cluster center, the final underwater biological target detection result is determined, achieving accurate detection of target underwater organisms in complex underwater environments.

[0171] In one example, a method for detecting and removing false detections of marine organisms based on a multimodal algorithm is provided, comprising the following steps:

[0172] Step 1: Determine the types of marine organisms that need to be detected, save the category names in text format, and collect a small number of pictures for each type of target and annotate them.

[0173] Specifically, the categories of marine organisms that need to be detected are defined in advance, and the category names are saved in text; a small number of pictures are collected for each type of target, including using crawler tools to collect relevant pictures on the Internet, or carrying underwater cameras to collect pictures of organisms in the ocean; for the collected pictures, a rectangular frame is used to mark the location of the target organism in the picture and different labels are assigned.

[0174] Step 2: For the text format category names obtained in step 1, a large language model is used to obtain a detailed description of each category.

[0175] Specifically, the large language model refers to any large language model based on deep learning, including but not limited to Llama, Qwen, ChatGPT, etc. The category text of the organism to be inspected is used as the input of the large language model to obtain a detailed description of each category.

[0176] Step 3: construct a multimodal detection model, load pre-trained parameters, and fine-tune the model by combining the pictures and category names collected in step 1 with the category descriptions obtained in step 2 to obtain a fine-tuned multimodal detection model.

[0177] Specifically, the multimodal detection model refers to any multimodal pre-trained detection model based on deep learning, including but not limited to GLIP, GroundingDino, YoloWorld, etc.

[0178] Load the pre-trained parameters of the multimodal model. The training data includes <image, category name, category description> triples. Figure 4 As shown, the image is encoded using the visual encoder of the multimodal model to obtain image features; the category name and category description are encoded using the text encoder of the multimodal model to obtain category name features and category description features, and the category name features and category description features are concatenated to obtain category text features; the image features and category text features are input into the image-text feature fusion module to obtain image-text fusion features; the image-text fusion features are sent to the detection head to obtain the target box classification loss and the target box position regression loss; and the back propagation algorithm is used to update the model parameters.

[0179] Step 4: Use the multimodal detection model obtained in step 3 to detect marine life in an actual environment and collect the falsely detected targets.

[0180] Specifically, the fine-tuned multimodal detection model is used to infer marine life images, detect predefined biological categories, and collect targets that have been misdetected.

[0181] Step 5: Use the image encoder of the multimodal detection model obtained in step 3 to encode the misdetected target obtained in step 4 to obtain the features of the misdetected target, and use the Kmeans method to cluster the features of the misdetected target to obtain several cluster centers.

[0182] Specifically, Figure 5 As shown, the visual encoder of the fine-tuned multimodal detection model is used to infer the misdetected target and obtain the features of the misdetected target; the Kmeans algorithm is used to cluster the features of the misdetected target to obtain and save several cluster centers.

[0183] Step 6: Combine the multimodal detection model obtained in step 3 with the cluster center of the misdetected target obtained in step 5, infer the image to be detected, calculate the similarity between the detected target and the cluster center, filter out targets with high similarity, and retain the remaining targets as the output of the detection model.

[0184] Specifically, Figure 6 As shown, the text encoder of the multimodal detection model is used to encode the category name of the target to be detected to obtain text features; the visual encoder of the multimodal detection model is used to encode the image to be detected to obtain image features; the image features and text features are simultaneously sent to the image-text feature fusion module of the multimodal detection model to obtain image-text fusion features; the fused features are sent to the detection head of the multimodal detection model to obtain the classification confidence of the target and the detection frame position of the target; the RoiAlign algorithm is used to combine the target detection frame position to obtain the target features in the image features; the feature similarity between the target features and the cluster center is calculated; if the target similarity and the cluster center similarity exceed the threshold, it means that this target is likely to be a misdetected target, and this target can be filtered at this time; if the target similarity and the cluster center similarity are lower than the threshold, the result is output.

[0185] The marine life detection and false detection removal method based on the multimodal algorithm provided in this example uses a pre-trained multimodal model. It only needs to collect a small amount of pictures to fine-tune the model to achieve the detection of marine life, saving labor costs. Further, the training of the multimodal model is optimized. In the training process, the large language model is combined to obtain the category description. Compared with the previous use of only category nouns as text training, it can provide richer text features, thereby improving the generalization ability of the model. Further, the false detection removal ability of the multimodal model is optimized. The multimodal detection algorithm itself does not have the ability to eliminate false detections. This method uses Kmeans clustering in the existing false detection data to obtain the false detection cluster center. In actual detection, the similarity between each target and the false detection cluster center is calculated, so that the false detection target can be filtered. Therefore, compared with the prior art, this example combines multimodal detection technology, large language model technology, clustering technology, etc., and can detect the target using only a small amount of data to train the model, and reduce the false detection of irrelevant targets.

[0186] Furthermore, based on the marine organism detection and false detection removal method based on the multimodal algorithm provided in this example, a specific embodiment is provided, including:

[0187] Step 1: Use Google search engine to find 100 pictures of beluga whales, killer whales, and blue whales, and label them;

[0188] Step 2, use the Qwen-1.8B model to describe the beluga whale, killer whale, and blue whale respectively, and save the text after the description;

[0189] Step 3: Load the pre-trained parameters of the GLIP-T model and train the model using data from beluga whales, killer whales, and blue whales.

[0190] Step 4: Infer the trained model in actual scenarios and collect false positives, such as dolphins, whale sharks, starfish, etc.

[0191] Step 5: Use GLIP-T's visual encoder to encode the misdetected targets collected in step 4 to obtain the features of the misdetected targets, and use the Kmeans clustering algorithm to cluster the features to obtain several cluster centers.

[0192] In step 6, the GLIP-T model obtained in step 3 and the cluster center obtained in step 5 are combined as the final detection and false detection removal model. The model is inferred in the actual scene, and finally beluga whales, killer whales, and blue whales can be detected, and false detections of targets such as dolphins, whale sharks, and starfish can be reduced.

[0193] In this embodiment, an underwater biological detection device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0194] This embodiment provides an underwater biological detection device, such as Figure 7 As shown, the device comprises:

[0195] The first acquisition module 701 is used to acquire the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected, and multiple underwater creature category misdetection cluster centers.

[0196] The first processing module 702 is used to input the name of the underwater creature category to be detected into the large language model for processing to obtain the description of the underwater creature category to be detected.

[0197] The second processing module 703 is used to input the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected into the target multimodal detection model for processing, so as to obtain multiple target visual features and multiple underwater creature initial detection results.

[0198] The calculation module 704 is used to calculate the similarity between each target visual feature and each underwater organism category misdetected cluster center to obtain multiple similarity values.

[0199] The comparison and determination module 705 is used to compare the multiple similarity values ​​with the preset similarity thresholds respectively, and determine multiple underwater organism target detection results according to the comparison results and multiple underwater organism initial detection results.

[0200] In some optional implementations, the first acquisition module 701 includes:

[0201] The acquisition submodule is used to obtain multiple historical false positive targets.

[0202] The input submodule is used to input multiple historical false detection targets into the visual encoder of the target multimodal detection model to obtain multiple historical false detection target features.

[0203] The analysis submodule is used to perform cluster analysis on multiple historical false positive target features to obtain multiple false positive cluster centers of underwater organism categories.

[0204] In some optional implementations, the acquisition submodule includes:

[0205] The acquisition unit is used to acquire a plurality of first underwater creature category names and a plurality of first underwater creature category pictures.

[0206] The processing unit is used to process the multiple first underwater creature category names and the multiple first underwater creature category pictures through the large language model and the target multimodal detection model to obtain multiple underwater creature detection results.

[0207] The determination unit is used to determine a plurality of historical false detection targets based on a plurality of underwater organism detection results.

[0208] In some optional embodiments, the device further comprises:

[0209] The second acquisition module is used to acquire a plurality of second underwater life category names and a preset number of second underwater life category pictures.

[0210] The third processing module is used to process multiple second underwater creature category names and a preset number of second underwater creature category pictures through a large language model and a preset target multimodal detection model to obtain a target frame classification loss and a target frame position regression loss.

[0211] The adjustment module is used to adjust the preset multimodal detection model using the target box classification loss and the target box position regression loss to obtain the target multimodal detection model.

[0212] In some optional implementations, the target multimodal detection model includes a text encoder, a visual encoder, a graphic feature fusion module and a detection head; the second processing module 703 includes:

[0213] The first processing submodule is used to process the name of the underwater creature category to be detected and the description of the underwater creature category to be detected through a text encoder to obtain multiple text features.

[0214] The second processing submodule is used to process the image of the underwater organism category to be detected through a visual encoder to obtain a plurality of initial visual features.

[0215] The third processing submodule is used to process the multiple text features and the multiple initial visual features through the image-text feature fusion module and the detection head respectively, so as to obtain multiple target classification confidences and multiple target detection frame positions.

[0216] The generation submodule is used to generate multiple underwater organism initial detection results based on multiple target classification confidences and multiple target detection box positions.

[0217] The fourth processing submodule is used to process the multiple initial visual features and the multiple target detection frame positions through a preset regional feature aggregation algorithm to obtain multiple target visual features.

[0218] In some optional implementations, the comparison and determination module 705 includes:

[0219] The comparison submodule is used to compare the multiple similarity values ​​with the preset similarity thresholds respectively.

[0220] The filtering submodule is used to filter the underwater organism initial detection results that do not meet the conditions among the multiple underwater organism initial detection results according to the comparison results, and obtain multiple underwater organism target detection results, wherein the underwater organism initial detection results that do not meet the conditions represent the detection results of underwater organisms whose similarity values ​​are greater than a preset similarity threshold.

[0221] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0222] The underwater biological detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0223] The embodiment of the present invention also provides a computer device having the above Figure 7 The underwater biological detection device shown.

[0224] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0225] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0226] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0227] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0228] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0229] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0230] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0231] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0232] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting underwater organisms, characterized in that: The method comprises: Obtain the name of the underwater organism category to be detected, the image of the underwater organism category to be detected, and multiple false positive cluster centers of the underwater organism category; Input the name of the underwater organism category to be detected into the large language model for processing to obtain a description of the underwater organism category to be detected; Input the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected into the target multimodal detection model for processing, so as to obtain multiple target visual features and multiple underwater creature initial detection results; Calculating the similarity between each of the target visual features and each of the underwater organism category misdetected cluster centers to obtain a plurality of similarity values; The multiple similarity values ​​are respectively compared with preset similarity thresholds, and multiple underwater organism target detection results are determined according to the comparison results and the multiple underwater organism initial detection results.

2. The method according to claim 1, characterized in that: Get the false positive cluster centers of multiple underwater creature categories, including: Obtain multiple historical false positive targets; Inputting the multiple historical false detection targets into the visual encoder of the target multimodal detection model to obtain multiple historical false detection target features; Cluster analysis is performed on the multiple historical false detection target features to obtain the multiple false detection cluster centers of the underwater organism categories.

3. The method according to claim 2, characterized in that Get multiple historical false positive targets, including: Obtain multiple first underwater creature category names and multiple first underwater creature category pictures; Processing the multiple first underwater creature category names and the multiple first underwater creature category pictures through the large language model and the target multimodal detection model to obtain multiple underwater creature detection results; The plurality of historical false detection targets are determined based on the plurality of underwater organism detection results.

4. The method according to claim 1, characterized in that The method further comprises: Acquire a plurality of second underwater creature category names and a preset number of second underwater creature category pictures; Processing the plurality of second underwater creature category names and the preset number of second underwater creature category pictures through the large language model and the preset target multimodal detection model to obtain a target frame classification loss and a target frame position regression loss; The preset multimodal detection model is adjusted using the target box classification loss and the target box position regression loss to obtain the target multimodal detection model.

5. The method according to claim 1, characterized in that The target multimodal detection model includes a text encoder, a visual encoder, a graphic feature fusion module and a detection head; The name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected are input into the target multimodal detection model for processing, and multiple target visual features and multiple underwater creature initial detection results are obtained, including: Processing the name of the underwater creature category to be detected and the description of the underwater creature category to be detected by the text encoder to obtain multiple text features; Processing the underwater organism category picture to be detected through the visual encoder to obtain a plurality of initial visual features; Processing the plurality of text features and the plurality of initial visual features through the image-text feature fusion module to obtain a plurality of image-text fusion features; The multiple image-text fusion features are processed by the detection head to obtain multiple target classification confidences and multiple target detection frame positions; Generating the plurality of underwater life initial detection results based on the plurality of target classification confidences and the plurality of target detection frame positions; The multiple initial visual features and the multiple target detection frame positions are processed by a preset regional feature aggregation algorithm to obtain the multiple target visual features.

6. The method according to claim 5, characterized in that The multiple similarity values ​​are respectively compared with the preset similarity thresholds, and multiple underwater organism target detection results are determined according to the comparison results and the multiple underwater organism initial detection results, including: Comparing the multiple similarity values ​​with the preset similarity threshold respectively; According to the comparison result, the underwater organism initial detection results that do not meet the conditions are filtered out from the multiple underwater organism initial detection results to obtain the multiple underwater organism target detection results, wherein the underwater organism initial detection results that do not meet the conditions represent the detection results of underwater organisms whose similarity values ​​are greater than the preset similarity threshold.

7. An underwater biological detection device, characterized in that: The device comprises: The first acquisition module is used to obtain the name of the underwater organism category to be detected, the picture of the underwater organism category to be detected, and multiple underwater organism category false detection cluster centers; The first processing module is used to input the name of the underwater creature category to be detected into the large language model for processing to obtain a description of the underwater creature category to be detected; The second processing module is used to input the name of the underwater creature category to be detected, the picture of the underwater creature category to be detected and the description of the underwater creature category to be detected into the target multimodal detection model for processing, so as to obtain multiple target visual features and multiple underwater creature initial detection results; A calculation module, used for calculating the similarity between each of the target visual features and each underwater organism category misdetected cluster center, to obtain a plurality of similarity values; The comparison and determination module is used to compare the multiple similarity values ​​with the preset similarity thresholds respectively, and determine multiple underwater organism target detection results according to the comparison results and the multiple underwater organism initial detection results.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the underwater organism detection method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the underwater organism detection method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the underwater organism detection method according to any one of claims 1 to 6.

Images