A target detection and identification method, device, equipment and medium in a marine environment

Abstract

The application discloses a target detection and identification method and device in a marine environment, equipment and a medium, relates to the technical field of target detection, and comprises the following steps: performing preset processing on an image dataset obtained by shooting in a marine environment to obtain a processed dataset; inputting the processed dataset into a backbone network of an initial model for feature extraction to obtain feature information; inputting the feature information into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information; and performing positioning training and classification training on a classification prediction network by using the fused feature information to obtain a trained marine image detection and identification model. The application improves the extraction capability of target feature information by designing a feature fusion network, enhances the multi-scale feature fusion capability of the network, makes the network more suitable for the actual underwater environment, improves the recognition efficiency and recognition accuracy, and reduces the algorithm power consumption.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method, apparatus, equipment and medium for target detection and identification in a marine environment. Background Technology

[0002] The development of marine resources and the balance of ecosystems play a vital role in human life, and many marine management strategies utilize remote sensing to detect marine life and habitats. In recent years, due to the widespread use of digital cameras deployed on autonomous underwater vehicles (AUVs) and unmanned underwater vehicles (UUVs), the availability of underwater imagery has increased exponentially. Therefore, deep learning-based marine target detection offers unprecedented research opportunities for many marine targets. Target detection in the marine environment has significant research value and application prospects, and can provide technical support for the monitoring, protection, and sustainable development of marine biological resources such as fish.

[0003] The task of object detection is to identify valuable objects in an image and determine their location and category. Besides image classification, object detection faces challenges such as: objects can appear anywhere in an image; objects can have various sizes; and objects can have various shapes.

[0004] Existing object detection algorithms can be divided into traditional object detection methods and deep learning-based object detection methods. Traditional object detection algorithms suffer from drawbacks such as difficulty in feature extraction and weak generalization ability, making it difficult to achieve ideal detection accuracy. With the continuous development of artificial intelligence, deep learning-based object detection algorithms have gradually become a new research focus. Deep learning-based object detection algorithms can be roughly divided into three categories: regression-based one-stage algorithms, such as YOLO (You Only Look Once, which can identify the category and location of objects in an image with only one look) and SSD (SingleShot MultiBox Detector, an object detection algorithm); classification-based two-stage algorithms, such as Faster R-CNN (region with CNN features); and end-to-end detection frameworks based on dense prediction, such as DETR (DEtectionTransformer, an end-to-end object detection network based on Transformer). Convolutional neural networks are the foundation of deep learning, using backpropagation for feedback learning and improving learning ability through automated feature extraction. Compared with traditional detection methods that rely on manual feature extraction, deep learning-based object detection methods achieve better learning results. As a branch of computer vision, deep learning-based object detection algorithms have been widely used in fields such as transportation, agriculture, and remote sensing.

[0005] Due to the complexity of the marine environment, objects vary in size, shape, and posture, and many targets are occluded or have incomplete information. Furthermore, light scattering and low contrast in the marine environment contribute to blurry images and indistinct features. Recent target detection algorithms applied to marine target detection first require image enhancement before detection and recognition, resulting in low efficiency, high computational consumption, and low recognition accuracy. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a target detection and identification method, apparatus, device, and medium in a marine environment, which is more adaptable to the actual underwater environment, improves identification efficiency and accuracy, and reduces computing power consumption. The specific solution is as follows:

[0007] Firstly, this application discloses a target detection and recognition method in a marine environment, including:

[0008] The acquired image dataset taken in the marine environment is subjected to pre-set processing to obtain the processed dataset;

[0009] The processed dataset is input into the backbone network of the initial model for feature extraction to obtain feature information;

[0010] The feature information is input into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information;

[0011] The fused feature information is used to train the classification prediction network for localization and classification, so as to obtain the trained marine image detection and recognition model.

[0012] Optionally, the acquired image dataset taken in the marine environment undergoes pre-processing to obtain a processed dataset, including:

[0013] Each image in the image dataset is labeled with the target species type using labeling software to obtain a labeled image dataset;

[0014] The labeled image dataset is divided into a training set, a validation set, and a test set according to a preset ratio.

[0015] Optionally, the step of inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information includes:

[0016] The training set is input into the four intermediate feature layers of the backbone network of the initial model to obtain the corresponding multi-scale output feature maps.

[0017] The multi-scale output feature maps are input into the corresponding horizontal convolutional layers to obtain the feature information corresponding to each output feature map.

[0018] Optionally, the step of inputting the feature information into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information includes:

[0019] The feature information is input into the feature fusion network, and the current execution count is set to 0; the feature information includes first feature information, second feature information, third feature information, and fourth feature information;

[0020] The feature information is subjected to horizontal convolution to obtain the corresponding convolutional feature map; wherein, the convolutional feature map includes a first convolutional feature map, a second convolutional feature map, a third convolutional feature map, and a fourth convolutional feature map;

[0021] The fourth feature information is downsampled to obtain the downsampled fourth feature information;

[0022] The second feature information is upsampled to obtain the upsampled second feature information;

[0023] The downsampled fourth feature information, the two third convolutional feature maps, and the upsampled second feature information are added together to obtain the first target feature map;

[0024] The third convolutional feature map is downsampled to obtain a downsampled third convolutional feature map;

[0025] The first convolutional feature map is upsampled to obtain the upsampled first convolutional feature map;

[0026] The second target feature map is obtained by adding the downsampled third convolutional feature map, the two second convolutional feature maps, and the upsampled first convolutional feature map.

[0027] The second convolutional feature map is downsampled to obtain the downsampled second convolutional feature map;

[0028] The second convolutional feature map after downsampling and the first convolutional feature map are added together to obtain the third target feature map;

[0029] Increment the current execution count by 1 to obtain the execution completion count, and determine whether the execution completion count has reached the preset execution count;

[0030] If the number of executions completed does not reach the preset number of executions, then the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map are determined as the feature information, and the process re-enters the step of inputting the feature information into the feature fusion network.

[0031] If the number of executions has reached the preset number of executions, then the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map are determined as output feature maps, and the fourth feature information is input into the feature enhancement region to obtain the enhanced fourth feature information;

[0032] The fourth convolutional feature map output from the last execution is determined as the target convolutional feature map, and the target convolutional feature map is added to the enhanced fourth feature information to obtain the fused feature information.

[0033] Optionally, after inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information, the method further includes:

[0034] A consistency supervision region is set on the feature information, and the region proposal network is combined with the output feature map through the consistency supervision region to obtain a preset number of ROI regions;

[0035] The ROI features corresponding to the ROI region are obtained using the ROI Align method.

[0036] Multiple classification and bounding box regression operations are performed on the ROI features to obtain the auxiliary loss.

[0037] Optionally, the step of using the fused feature information to perform localization training and classification training on the classification prediction network includes:

[0038] The fused feature information is input into the ClassNet and BoxNet classes of the classification prediction network for localization and classification training, respectively.

[0039] Optionally, after using the fused feature information to perform localization training and classification training on the classification prediction network, the method further includes:

[0040] Determine whether the current training iterations have reached the preset number of iterations;

[0041] If the current number of training iterations has not reached the preset number of iterations, then the process of inputting the processed dataset into the backbone network of the initial model for feature extraction is repeated to obtain feature information.

[0042] If the current number of training iterations has reached the preset number, then determine whether the current loss value has converged;

[0043] If the current loss value has converged, then the model obtained from the last training is determined to be the trained ocean image detection and recognition model.

[0044] Secondly, this application discloses a target detection and identification device in a marine environment, comprising:

[0045] The image dataset processing module is used to perform pre-set processing on the acquired image dataset taken in the marine environment to obtain the processed dataset;

[0046] The feature extraction module is used to input the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information;

[0047] The feature fusion module is used to input the feature information into the feature fusion network to perform a preset feature fusion operation to obtain fused feature information;

[0048] The classification prediction module is used to perform localization training and classification training on the classification prediction network using the fused feature information to obtain the trained marine image detection and recognition model.

[0049] Thirdly, this application discloses an electronic device, comprising:

[0050] Memory, used to store computer programs;

[0051] A processor is used to execute the computer program to implement the steps of the target detection and recognition method in a marine environment as disclosed above.

[0052] Fourthly, this application discloses a computer-readable medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the target detection and recognition method in a marine environment as disclosed above.

[0053] As can be seen, this application provides a target detection and recognition method in a marine environment, comprising: pre-processing an image dataset acquired in a marine environment to obtain a processed dataset; inputting the processed dataset into the backbone network of an initial model for feature extraction to obtain feature information; inputting the feature information into a feature fusion network to perform a pre-defined feature fusion operation to obtain fused feature information; and using the fused feature information to train a classification prediction network for localization and classification to obtain a trained marine image detection and recognition model. Therefore, this application constructs a marine image detection and recognition model comprising a backbone network, a feature fusion network, and a classification prediction network. By designing a feature fusion network, it improves the ability to extract feature information of targets, enhances the network's multi-scale feature fusion capability, makes the network more adaptable to the actual underwater environment, improves recognition efficiency and accuracy, and reduces computational consumption. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 This is a flowchart of a target detection and recognition method in a marine environment disclosed in this application;

[0056] Figure 2 This is a flowchart of a specific target detection and recognition method in a marine environment disclosed in this application;

[0057] Figure 3 This is a flowchart of a specific target detection and recognition method in a marine environment disclosed in this application;

[0058] Figure 4 This is a schematic diagram of the overall structure of the target detection and recognition model disclosed in this application;

[0059] Figure 5 This is a schematic diagram of the structure of FasterNet, the backbone network disclosed in this application.

[0060] Figure 6 This is a schematic diagram of the enhanced bidirectional feature pyramid network (AWBiFPN) structure disclosed in this application;

[0061] Figure 7 This is a schematic diagram of the residual feature enhancement module disclosed in this application;

[0062] Figure 8 This is a schematic diagram of the classification prediction network disclosed in this application for classifying and locating target objects.

[0063] Figure 9 This is a simplified structural block diagram of the marine biological target identification device disclosed in this application;

[0064] Figure 10 This is a schematic diagram of the target detection and identification device in the marine environment provided in this application;

[0065] Figure 11 This application provides a structural diagram of an electronic device. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Currently, due to the complexity of the marine environment, objects of various sizes, shapes, and postures, as well as numerous occluded and incomplete targets, coupled with light scattering and low contrast, images often appear blurry and feature-unclear. Recent target detection algorithms applied to marine target detection first require image enhancement before detection and recognition, resulting in low efficiency, high computational consumption, and low recognition accuracy. Therefore, this application provides a target detection and recognition method for marine environments that is more adaptable to actual underwater conditions, improving recognition efficiency and accuracy while reducing computational consumption.

[0068] This invention discloses a target detection and recognition method in a marine environment. (See also...) Figure 1 As shown, the method includes:

[0069] Step S11: Perform pre-processing on the acquired image dataset taken in the marine environment to obtain the processed dataset.

[0070] In this embodiment, the acquired image dataset taken in a marine environment undergoes pre-processing to obtain a processed dataset. Specifically, each image in the image dataset is labeled with the target species type using labeling software to obtain a labeled image dataset; the labeled image dataset is then divided into a training set, a validation set, and a test set according to a preset ratio. That is, several image datasets taken in a marine environment are acquired, the target species are labeled using labeling software, and the datasets are divided into training, validation, and test sets.

[0071] Understandably, the downloaded marine life dataset was manually labeled using standard software. The dataset format was the classic COCO (Common Objects in Context) target detection dataset, used for image recognition. It was divided into training, validation, and test sets in a 7:2:1 ratio (this ratio can be adjusted depending on the situation). For example, marine object datasets were obtained from multiple AI detection platforms and marine robot imaging platforms in different scenarios. After filtering and removing images without targets, a total of 12,000 valid images were retained.

[0072] Step S12: Input the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information.

[0073] In this embodiment, the acquired image dataset taken in the marine environment undergoes pre-processing to obtain a processed dataset. This processed dataset is then input into the backbone network of the initial model for feature extraction to obtain feature information. Specifically, the training set is input into the four intermediate feature layers of the backbone network of the initial model to obtain corresponding multi-scale output feature maps. These multi-scale output feature maps are then input into corresponding horizontal convolutional layers to obtain the feature information corresponding to each output feature map. It is understood that, as... Figure 2 As shown, images from several marine environments were acquired, and target classes were labeled using labeling software. The dataset format was COCO, and the labeled images were divided into training, validation, and test sets. A deep learning target detection algorithm model was built based on the FasterNet network model. The algorithm model includes three parts: a backbone network, a feature fusion network, and a classification prediction network. The backbone network described in this invention is the FasterNet network, used for preliminary extraction of image features. The designed feature fusion network is used to further fuse and enhance multi-scale feature information of target objects, textures, and backgrounds. The classification prediction network is used to locate and classify target information in the image. The training parameters of the algorithm model were set, and the model was trained using the training set, so that the model parameters were continuously optimized during the training process. The images from the test set were input into the trained model, and the detection results were output for verification.

[0074] Before inputting the processed dataset into the backbone network of the initial model for feature extraction, the training parameters of the initial algorithm model are set. For example, the training period is set to 12, the batch size is set to 4, and the initial learning rate is set to 0.001, where the learning rate uses the warm-up learning rate method.

[0075] Understandably, the acquired training set is input into the initial model. First, it passes through a backbone network, which contains multiple layers. Each batch of training images is input into the corresponding layer, and the backbone network obtains corresponding multi-scale feature maps (i.e., layers P1, P2, P3, and P4). The number of images in each batch depends on the batch size. Then, the obtained multi-scale feature maps are passed through four horizontal 1x1 convolutional layers to obtain the interdependencies between channels and reduce the number of channels, generating layers M1, M2, M3, and M4. At this point, layers M1, M2, M3, and M4 contain the feature information corresponding to the four training images input in the current batch.

[0076] Step S13: Input the feature information into the feature fusion network to perform a preset feature fusion operation to obtain the fused feature information.

[0077] In this embodiment, the processed dataset is input into the backbone network of the initial model for feature extraction. After obtaining feature information, the feature information is input into the feature fusion network to perform a preset feature fusion operation to obtain fused feature information. It can be understood that layers M1, M2, M3, and M4 are used as new feature layers and input into the feature fusion network (i.e., the enhanced bidirectional feature pyramid network AWBiFPN). After performing the preset feature fusion operation in the feature fusion network, the output of the feature fusion network is layers C1, C2, C3, and C4, which are the fused feature information.

[0078] Step S14: Use the fused feature information to perform localization training and classification training on the classification prediction network to obtain the trained marine image detection and recognition model.

[0079] In this embodiment, the feature information is input into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information. Then, the fused feature information is used to train the classification prediction network for localization and classification, resulting in a trained marine image detection and recognition model. It can be understood that the fused feature information is input into the ClassNet and BoxNet classes of the classification prediction network for corresponding localization and classification training. Layers C1, C2, C3, and C4 are input into the classification prediction network for target localization and classification. Simultaneously, the three networks are trained multiple times according to a set training period, allowing the model parameters to be continuously optimized during training.

[0080] Furthermore, the fused feature information is used to train the classification prediction network for localization and classification to obtain a trained marine image detection and recognition model. Then, the images of the test set are input into the trained model, and the detection results are output for verification.

[0081] This invention provides a target detection and recognition method for marine environments. The construction of a feature fusion network in this invention improves the neural network's ability to extract feature information from targets, especially stacked targets, small targets, and camouflaged targets, thus improving recognition accuracy. Furthermore, the processing results of the feature fusion network are used in an enhanced bidirectional feature pyramid network (AWBiFPN), enhancing the network's multi-scale feature fusion capabilities and making the network more adaptable to real-world underwater environments. Specifically, since the top feature map has rich semantic information, a residual feature enhancement module can enrich the semantic representation of the top-level features. Meanwhile, the bottom feature map has high resolution, and a consistency supervision module can reduce the semantic gap between multi-scale features, increasing the module's convergence speed, reducing training resources, and further improving detection accuracy and efficiency.

[0082] As can be seen, this application provides a target detection and recognition method in a marine environment, comprising: pre-processing an image dataset acquired in a marine environment to obtain a processed dataset; inputting the processed dataset into the backbone network of an initial model for feature extraction to obtain feature information; inputting the feature information into a feature fusion network to perform a pre-defined feature fusion operation to obtain fused feature information; and using the fused feature information to train a classification prediction network for localization and classification to obtain a trained marine image detection and recognition model. Therefore, this application constructs a marine image detection and recognition model comprising a backbone network, a feature fusion network, and a classification prediction network. By designing a feature fusion network, it improves the ability to extract feature information of targets, enhances the network's multi-scale feature fusion capability, makes the network more adaptable to the actual underwater environment, improves recognition efficiency and accuracy, and reduces computational consumption.

[0083] See Figure 3 As shown, this embodiment of the invention discloses a target detection and recognition method in a marine environment. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0084] Step S21: Perform pre-processing on the acquired image dataset taken in the marine environment to obtain the processed dataset.

[0085] Step S22: Input the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information.

[0086] In this embodiment, a deep learning-based object detection algorithm model is built based on the Fasternet network model. The algorithm model comprises three parts: a backbone (feature extraction) network, a feature fusion network, and a classification and prediction network. The overall structure of the model is as follows: Figure 4 As shown:

[0087] Backbone (Feature Extraction) Network: The backbone (feature extraction) network performs initial feature extraction from the image. The FasterNet network consists of four stacked blocks, as shown in the diagram. Figure 5 As shown, four multi-scale feature layers can be obtained through the backbone (feature extraction) network, of which features P1, P2, P3, and P4 are processed in the next step.

[0088] Feature fusion network: The feature fusion network further extracts the feature information of the target object's details, texture, and background. The specific structure is as follows: Figure 6 As shown. The specific structure and processing of the feature fusion network are as follows: The intermediate feature layers P1, P2, P3, and P4 of the backbone feature extraction network are used as inputs to the feature fusion network. Before input, layers P1, P2, P3, and P4 are respectively processed by four horizontal 1X1 convolutional layers to obtain the interdependence between channels and reduce the number of channels, thereby generating layers M1, M2, M3, and M4. Layers M1, M2, M3, and M4 are then used as new feature layers and input into the feature fusion network, namely the enhanced bidirectional feature pyramid network AWBiFPN. The output of the feature fusion network is layers C1, C2, C3, and C4. Finally, layers C1, C2, C3, and C4 are input into the classification prediction network for target localization and classification.

[0089] Step S23: Set a consistency supervision region on the feature information, and combine the region proposal network with the output feature map through the consistency supervision region to obtain a preset number of ROI regions.

[0090] In this embodiment, the processed dataset is input into the backbone network of the initial model for feature extraction. After obtaining feature information, a consistency supervision region is set on the feature information. That is, a consistency supervision module is added to layers M1, M2, M3, and M4 to reduce the semantic information gap between features at each layer. The region proposal network is then combined with the output feature map through the consistency supervision region to obtain a preset number of ROIs (Regions of Interest). It can be understood that the enhanced bidirectional feature pyramid network AWBiFPN adds a consistency supervision module and a residual feature enhancement module to the BiFPN (Bidirectional Feature Pyramid Network). The consistency supervision module combines the Region Proposal Network (RPN) with the feature pyramid layers C1, C2, C3, and C4 at layers M1, M2, M3, and M4 to generate multiple ROI regions.

[0091] Step S24: Use the ROI Align method to obtain the ROI features corresponding to the ROI region, and perform multi-classification and bounding box regression operations on the ROI features to obtain auxiliary loss.

[0092] In this embodiment, the Region Proposal Network (ROI) is combined with the output feature map through the consistency supervision region to obtain a preset number of ROI regions. Then, the ROI Align method is used to obtain the ROI features corresponding to the ROI regions, and multi-classification and bounding box regression operations are performed on the ROI features to obtain an auxiliary loss. That is, the ROI Align method is used to obtain the ROI features at each level of layers M1, M2, M3, and M4, and then after performing multi-classification and bounding box regression on these features, an auxiliary loss is generated.

[0093] Understandably, different feature maps can learn similar semantic information by sharing parameters of these classification and regression heads across levels. To balance the auxiliary loss of consistent supervision with the original loss, the following loss function was designed:

[0094]

[0095] Where L cls,M L loc,M It is the objective function corresponding to the auxiliary losses on layers M1, M2, M3, and M4, while L cls,c L loc,c This is the original loss function for layers C1, C2, C3, and C4. The predictions for the intermediate layers and the final pyramid layer are shown in p. M d M In p and d, λ represents the weight used to balance the auxiliary loss and the original loss, and β represents the loss used to stabilize classification and localization. Definition The formula is as follows:

[0096]

[0097] In the process of fusing the top feature map P4 with other hierarchical features via a bidirectional path, there is a possibility of semantic information loss due to the inevitable reduction in the number of channels. Furthermore, this method only contains single-scale contextual information, which may lead to incompatibility with the resulting features at other levels, thus limiting its effectiveness. This invention designs a residual feature enhancement module to improve the representation of the top-level features, such as... Figure 7 As shown, residual design is used to generate different spatial context information, which is then merged with C4. To create multiple context features, a fixed-size adaptive pooling is performed on the P4 feature map of size S, resulting in (α1×S, α2×S, ... α) feature maps of different sizes. nThe system extracts multiple contextual features from the S x S matrix, then convolves these contextual features with a 1x1 matrix to reduce the feature channel dimension to 256, and then upsamples them to S using bilinear interpolation. To better integrate these features, the present invention designs an adaptive spatial fusion (ASF) method to effectively integrate them, adding M5 to the output feature C4 for the final prediction.

[0098] Step S25: Input the feature information into the feature fusion network to perform a preset feature fusion operation to obtain the fused feature information.

[0099] In this embodiment, the feature information is input into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information. Specifically, the feature information is input into the feature fusion network, and the current execution count is set to 0; the feature information includes first feature information, second feature information, third feature information, and fourth feature information; the feature information is subjected to horizontal convolution to obtain corresponding convolutional feature maps; wherein, the convolutional feature maps include a first convolutional feature map (M1_td), a second convolutional feature map (M2_td), a third convolutional feature map (M3_td), and a fourth convolutional feature map (M4_td, M4_out); the fourth feature information is downsampled to obtain downsampled fourth feature information; the second feature information is upsampled to obtain upsampled fourth feature information. The second feature information (M2_out) is obtained by adding the downsampled fourth feature information, the two third convolutional feature maps, and the upsampled second feature information (M2_out). The third convolutional feature map (M3_td) is downsampled to obtain a downsampled third convolutional feature map. The first convolutional feature map is upsampled to obtain an upsampled first convolutional feature map (M1_out). The downsampled third convolutional feature map, the two second convolutional feature maps (M2_td), and the upsampled first convolutional feature map are added to obtain a second target feature map (M2_out). The second convolutional feature map (M2_td) is downsampled to obtain a downsampled second convolutional feature map; the downsampled second convolutional feature map and the first convolutional feature map are added together to obtain a third target feature map (M1_out); the current execution count is incremented by 1 to obtain the execution completion count, and it is determined whether the execution completion count has reached a preset execution count; if the execution completion count has not reached the preset execution count, the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map are identified as the feature information, and the process re-enters the step of inputting the feature information into the feature fusion network, i.e., the above steps. The feature maps M4_out, M3_out, M2_out, and M1_out are repeatedly input (the number of inputs is customizable, for example, 3 times) to enter the step of inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information. The final outputs are C1, C2, C3, and C4. If the number of executions has reached the preset number of executions, the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map are determined as output feature maps (C1, C2, C3, C4), and the fourth feature information is input into the feature enhancement region to obtain the enhanced fourth feature information.The fourth convolutional feature map output from the last execution is determined as the target convolutional feature map. This target convolutional feature map is then added to the enhanced fourth feature information to obtain the fused feature information. Specifically, M4 (M4 being the top-level feature) is processed by the residual feature enhancement module to obtain M5, and M5 is then added to C4 to obtain the fused feature information.

[0100] Step S26: Use the fused feature information to perform localization training and classification training on the classification prediction network to obtain the trained marine image detection and recognition model.

[0101] In this embodiment, after using the fused feature information to perform localization and classification training on the classification prediction network to obtain the trained marine image detection and recognition model, it is determined whether the current training iterations have reached a preset number of iterations. If the current training iterations have not reached the preset number of iterations, the process re-enters the step of inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information. If the current training iterations have reached the preset number of iterations, it is determined whether the current loss value has converged. If the current loss value has converged, the model obtained from the last training iteration is determined to be the trained marine image detection and recognition model.

[0102] It is understandable that the classification prediction network consists of the class prediction network ClassNet and the bounding box prediction network BoxNet, such as... Figure 8 As shown, object classes and bounding box locations are generated respectively. The ClassNet class prediction network consists of two 256-channel convolutions and one num_anchors x num_classes convolution, used to predict the class corresponding to each predicted box at each grid point on the feature layer. The num_anchors refers to the number of prior boxes in the feature layer, and the num_classes refers to the total number of target classes detected by the network. The ClassNet used by each feature layer is the same ClassNet, and the BoxNet used by each feature layer is the same BoxNet.

[0103] Furthermore, when training the completed algorithm model, training can be performed on a Linux system, requiring an NVIDIA GTX 3090 (Giga Texel Shader eXtreme) GPU (NVIDIA Corporation) and 24GB of RAM (Random Access Memory). The software platform includes mmdetection 2.0. Label annotations are set, and the label information for the training and validation sets is configured based on the category information. Training parameters are set in the model training file `train.py`, with the training epochs set to 12, the batch size (number of training samples) set to 4, and the initial learning rate set to 0.005 (using a warm-up learning rate). Then, running the training file `train.py` starts training. After 12 epochs, the trained weights file will be generated in the corresponding weights folder.

[0104] After training, the trained model is tested. The prediction results require the prediction file `predict.py`. First, the model path `model-path` and the class path `classes_path` need to be modified. `model-path` should point to the trained weight file, and `classes_path` should point to the corresponding text file for the detection classes. After completing these steps, `predict.py` can be run to perform detection. Inputting the path to the test image will output the detection results.

[0105] Understandably, after the marine life target detection algorithm model is built, the steps in actual use are as follows: acquire marine life images; input the acquired images into the detection algorithm model built using the method described above; and obtain the types of marine life targets after model processing. In one specific implementation, such as... Figure 9As shown, this invention also provides a device for marine biological target identification. The device includes at least one processor and at least one memory, as well as a communication interface and an internal bus. The memory stores a program for a pre-built detection algorithm model. When the processor executes the program stored in the memory, it can realize the detection and identification of marine biological targets. The acquisition device is an underwater high-definition camera with a light source. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. The bus in the accompanying drawings is not limited to only one bus or one type of bus. The memory may include high-speed RAM memory, and may also include non-volatile memory NVM (node ​​version manager), such as at least one disk storage device, or it can be a USB flash drive, portable hard drive, read-only memory, disk, or optical disk, etc.

[0106] For details regarding step S21, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0107] As can be seen, this embodiment of the application preprocesses the acquired image dataset taken in the marine environment to obtain a processed dataset; inputs the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information; sets a consistency supervision region on the feature information, and combines the region proposal network with the output feature map through the consistency supervision region to obtain a preset number of ROI regions; uses the ROI Align method to obtain the ROI features corresponding to the ROI regions; performs multi-classification and bounding box regression operations on the ROI features to obtain auxiliary loss; inputs the feature information into the feature fusion network to perform a preset feature fusion operation to obtain fused feature information; uses the fused feature information to perform localization training and classification training on the classification prediction network to obtain a trained marine image detection and recognition model, which is more adapted to the actual underwater environment, improves recognition efficiency and accuracy, and reduces computing power consumption.

[0108] See Figure 10 As shown in the embodiments, this application also discloses a target detection and recognition device in a marine environment, including:

[0109] The image dataset processing module 11 is used to perform preset processing on the acquired image dataset taken in the marine environment to obtain the processed dataset;

[0110] The feature extraction module 12 is used to input the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information;

[0111] The feature fusion module 13 is used to input the feature information into the feature fusion network to perform a preset feature fusion operation to obtain the fused feature information;

[0112] The classification prediction module 14 is used to perform localization training and classification training on the classification prediction network using the fused feature information to obtain the trained marine image detection and recognition model.

[0113] As can be seen, this application includes: pre-processing an image dataset acquired in a marine environment to obtain a processed dataset; inputting the processed dataset into the backbone network of an initial model for feature extraction to obtain feature information; inputting the feature information into a feature fusion network to perform a pre-defined feature fusion operation to obtain fused feature information; and using the fused feature information to train a classification prediction network for localization and classification to obtain a trained marine image detection and recognition model. Therefore, this application constructs a marine image detection and recognition model comprising a backbone network, a feature fusion network, and a classification prediction network. By designing a feature fusion network, it improves the ability to extract feature information of target objects, enhances the network's multi-scale feature fusion capability, makes the network more adaptable to the actual underwater environment, improves recognition efficiency and accuracy, and reduces computational consumption.

[0114] In some specific embodiments, the image dataset processing module 11 specifically includes:

[0115] The type labeling unit is used to label each image in the image dataset with the target species type using labeling software to obtain a labeled image dataset;

[0116] The dataset partitioning unit is used to divide the labeled image dataset into a training set, a validation set, and a test set according to a preset ratio.

[0117] In some specific embodiments, the feature extraction module 12 specifically includes:

[0118] The output feature map acquisition unit is used to input the training set into the four intermediate feature layers of the backbone network of the initial model to obtain the corresponding multi-scale output feature maps.

[0119] The feature information acquisition unit is used to input the multi-scale output feature maps into the corresponding horizontal convolutional layers to obtain the feature information corresponding to each output feature map;

[0120] A consistency supervision region setting unit is used to set a consistency supervision region on the feature information;

[0121] The ROI region acquisition unit is used to combine the region proposal network with the output feature map through the consistency supervision region to obtain a preset number of ROI regions;

[0122] The ROI feature acquisition unit is used to acquire the ROI features corresponding to the ROI region using the ROI Align method.

[0123] The auxiliary loss acquisition unit is used to perform multiple classification operations and bounding box regression operations on the ROI features to obtain the auxiliary loss.

[0124] In some specific embodiments, the feature fusion module 13 specifically includes:

[0125] The first feature information input unit is used to input the feature information into the feature fusion network; the feature information includes first feature information, second feature information, third feature information, and fourth feature information.

[0126] The current execution count setting unit is used to set the current execution count to 0;

[0127] A lateral convolutional unit is used to perform lateral convolution on the feature information to obtain a corresponding convolutional feature map; wherein, the convolutional feature map includes a first convolutional feature map, a second convolutional feature map, a third convolutional feature map, and a fourth convolutional feature map;

[0128] The first downsampling unit is used to downsample the fourth feature information to obtain the downsampled fourth feature information;

[0129] The first upsampling unit is used to upsample the second feature information to obtain the upsampled second feature information;

[0130] The first target feature map acquisition unit is used to add the downsampled fourth feature information, the two third convolutional feature maps, and the upsampled second feature information to obtain the first target feature map;

[0131] The second downsampling unit is used to downsample the third convolutional feature map to obtain the downsampled third convolutional feature map.

[0132] The second upsampling unit is used to upsample the first convolutional feature map to obtain the upsampled first convolutional feature map.

[0133] The second target feature map acquisition unit is used to add the downsampled third convolutional feature map, the two second convolutional feature maps, and the upsampled first convolutional feature map to obtain the second target feature map;

[0134] The third downsampling unit is used to downsample the second convolutional feature map to obtain the downsampled second convolutional feature map.

[0135] The third target feature map acquisition unit is used to add the downsampled second convolutional feature map and the first convolutional feature map to obtain the third target feature map;

[0136] The execution completion count acquisition unit is used to increment the current execution count by 1 to obtain the execution completion count;

[0137] The execution completion count determination unit is used to determine whether the execution completion count has reached the preset execution count;

[0138] The feature information determination unit is used to determine the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map as the feature information if the number of execution completions does not reach the preset number of executions.

[0139] The second feature information input unit is used to re-enter the step of inputting the feature information into the feature fusion network;

[0140] The output feature map determination unit is used to determine the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map as output feature maps if the number of execution completions has reached the preset number of executions.

[0141] The fourth feature information enhancement unit is used to input the fourth feature information into the feature enhancement region to obtain the enhanced fourth feature information;

[0142] The target convolutional feature map determination unit is used to determine the fourth convolutional feature map output from the last execution as the target convolutional feature map;

[0143] The fused feature information acquisition unit is used to add the target convolutional feature map to the enhanced fourth feature information to obtain the fused feature information.

[0144] In some specific embodiments, the classification prediction module 14 specifically includes:

[0145] The fused feature information input unit is used to input the fused feature information into the ClassNet class prediction network and the BoxNet bounding box prediction network of the classification prediction network for corresponding localization training and classification training.

[0146] The pre-training count judgment unit is used to determine whether the current training count has reached the preset count;

[0147] The processed dataset input unit is used to re-enter the step of inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information if the current training count has not reached the preset count.

[0148] A convergence determination unit is used to determine whether the current loss value has converged if the current training iterations have reached the preset number of iterations.

[0149] The model determination unit is used to determine, if the current loss value has converged, the model obtained from the last training is the trained marine image detection and recognition model.

[0150] Furthermore, embodiments of this application also provide an electronic device. Figure 11 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0151] Figure 11 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the target detection and recognition method in the marine environment disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0152] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0153] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0154] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the target detection and recognition method in a marine environment as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0155] Furthermore, this application also discloses a medium storing a computer program, which, when loaded and executed by a processor, implements the target detection and recognition method steps in the marine environment disclosed in any of the foregoing embodiments.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0157] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The above provides a detailed description of a target detection and identification method, apparatus, device, and medium in a marine environment provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A target detection and recognition method in a marine environment, characterized in that, include: The acquired image dataset taken in the marine environment is subjected to pre-set processing to obtain the processed dataset; The processed dataset is input into the backbone network of the initial model for feature extraction to obtain feature information; The feature information is input into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information; The fused feature information is used to train the classification prediction network for localization and classification, so as to obtain the trained marine image detection and recognition model. The step of inputting the feature information into a feature fusion network to perform a preset feature fusion operation to obtain fused feature information includes: The feature information is input into the feature fusion network, and the current execution count is set to 0; the feature information includes first feature information, second feature information, third feature information, and fourth feature information; The feature information is subjected to horizontal convolution to obtain the corresponding convolutional feature map; wherein, the convolutional feature map includes a first convolutional feature map, a second convolutional feature map, a third convolutional feature map, and a fourth convolutional feature map; The fourth feature information is downsampled to obtain the downsampled fourth feature information; The second feature information is upsampled to obtain the upsampled second feature information; The downsampled fourth feature information, the third convolutional feature map, and the upsampled second feature information are added together to obtain the first target feature map; The third convolutional feature map is downsampled to obtain a downsampled third convolutional feature map; The first convolutional feature map is upsampled to obtain the upsampled first convolutional feature map; The second target feature map is obtained by adding the downsampled third convolutional feature map, the second convolutional feature map, and the upsampled first convolutional feature map. The second convolutional feature map is downsampled to obtain the downsampled second convolutional feature map; The second convolutional feature map after downsampling and the first convolutional feature map are added together to obtain the third target feature map; Increment the current execution count by 1 to obtain the execution completion count, and determine whether the execution completion count has reached the preset execution count; If the number of executions completed does not reach the preset number of executions, then the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map are determined as the feature information, and the process re-enters the step of inputting the feature information into the feature fusion network. If the number of executions has reached the preset number of executions, then the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map are determined as output feature maps, and the fourth feature information is input into the feature enhancement region to obtain the enhanced fourth feature information; The fourth convolutional feature map output from the last execution is determined as the target convolutional feature map, and the target convolutional feature map is added to the enhanced fourth feature information to obtain the fused feature information.

2. The target detection and recognition method in a marine environment according to claim 1, characterized in that, The process of pre-processing the acquired image dataset taken in the marine environment to obtain a processed dataset includes: Each image in the image dataset is labeled with the target species type using labeling software to obtain a labeled image dataset; The labeled image dataset is divided into a training set, a validation set, and a test set according to a preset ratio.

3. The target detection and recognition method in a marine environment according to claim 2, characterized in that, The step of inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information includes: The training set is input into the four intermediate feature layers of the backbone network of the initial model to obtain the corresponding multi-scale output feature maps. The multi-scale output feature maps are input into the corresponding horizontal convolutional layers to obtain the feature information corresponding to each output feature map.

4. The target detection and recognition method in a marine environment according to claim 1, characterized in that, After inputting the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information, the method further includes: A consistency supervision region is set on the feature information, and the region proposal network is combined with the output feature map through the consistency supervision region to obtain a preset number of ROI regions; The ROI features corresponding to the ROI region are obtained using the ROI Align method. Multiple classification and bounding box regression operations are performed on the ROI features to obtain the auxiliary loss.

5. The target detection and recognition method in a marine environment according to claim 1, characterized in that, The step of using the fused feature information to perform localization training and classification training on the classification prediction network includes: The fused feature information is input into the ClassNet and BoxNet classes of the classification prediction network for localization and classification training, respectively.

6. The target detection and recognition method in a marine environment according to any one of claims 1 to 5, characterized in that, After using the fused feature information to perform localization training and classification training on the classification prediction network, the method further includes: Determine whether the current training iterations have reached the preset number of iterations; If the current number of training iterations has not reached the preset number of iterations, then the process of inputting the processed dataset into the backbone network of the initial model for feature extraction is repeated to obtain feature information. If the current number of training iterations has reached the preset number, then determine whether the current loss value has converged; If the current loss value has converged, then the model obtained from the last training is determined to be the trained ocean image detection and recognition model.

7. A target detection and identification device for a marine environment, characterized in that, include: The image dataset processing module is used to perform pre-set processing on the acquired image dataset taken in the marine environment to obtain the processed dataset; The feature extraction module is used to input the processed dataset into the backbone network of the initial model for feature extraction to obtain feature information; The feature fusion module is used to input the feature information into the feature fusion network to perform a preset feature fusion operation to obtain fused feature information; The classification prediction module is used to perform localization training and classification training on the classification prediction network using the fused feature information, so as to obtain the trained marine image detection and recognition model. The feature fusion module includes: The first feature information input unit is used to input the feature information into the feature fusion network; the feature information includes first feature information, second feature information, third feature information, and fourth feature information. The current execution count setting unit is used to set the current execution count to 0; A lateral convolutional unit is used to perform lateral convolution on the feature information to obtain a corresponding convolutional feature map; wherein, the convolutional feature map includes a first convolutional feature map, a second convolutional feature map, a third convolutional feature map, and a fourth convolutional feature map; The first downsampling unit is used to downsample the fourth feature information to obtain the downsampled fourth feature information; The first upsampling unit is used to upsample the second feature information to obtain the upsampled second feature information; The first target feature map acquisition unit is used to add the downsampled fourth feature information, the third convolutional feature map, and the upsampled second feature information to obtain the first target feature map; The second downsampling unit is used to downsample the third convolutional feature map to obtain the downsampled third convolutional feature map. The second upsampling unit is used to upsample the first convolutional feature map to obtain the upsampled first convolutional feature map. The second target feature map acquisition unit is used to add the downsampled third convolutional feature map, the second convolutional feature map, and the upsampled first convolutional feature map to obtain the second target feature map; The third downsampling unit is used to downsample the second convolutional feature map to obtain the downsampled second convolutional feature map. The third target feature map acquisition unit is used to add the downsampled second convolutional feature map and the first convolutional feature map to obtain the third target feature map; The execution completion count acquisition unit is used to increment the current execution count by 1 to obtain the execution completion count; The execution completion count determination unit is used to determine whether the execution completion count has reached the preset execution count; The feature information determination unit is used to determine the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map as the feature information if the number of execution completions does not reach the preset number of executions. The second feature information input unit is used to re-enter the step of inputting the feature information into the feature fusion network; The output feature map determination unit is used to determine the first target feature map, the second target feature map, the third target feature map, and the fourth convolutional feature map as output feature maps if the number of execution completions has reached the preset number of executions. The fourth feature information enhancement unit is used to input the fourth feature information into the feature enhancement region to obtain the enhanced fourth feature information; The target convolutional feature map determination unit is used to determine the fourth convolutional feature map output from the last execution as the target convolutional feature map; The fused feature information acquisition unit is used to add the target convolutional feature map to the enhanced fourth feature information to obtain the fused feature information.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the target detection and recognition method in a marine environment as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the target detection and recognition method in a marine environment as described in any one of claims 1 to 6.

Images