A domain-adaptive marine benthic organism detection method based on consistent light field and color distribution

Abstract

The present invention discloses a domain-adaptive marine benthic organism detection method based on consistent light field and color distribution, comprising the following steps: obtaining paired degraded reference marine benthic organism images containing any one or more of sea urchins, scallops, starfish, and sea cucumbers to construct a data set; dividing the data set to obtain a training set and a test set; constructing a domain-adaptive detection model based on consistent light field and color distribution; training the domain-adaptive detection model based on consistent light field and color distribution based on the training set data to obtain a trained domain-adaptive detection model based on consistent light field and color distribution; and inputting the test set data into the trained domain-adaptive detection model based on consistent light field and color distribution to implement detection of any one or more of sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

Description

Technical Field

[0001] The present invention belongs to the field of underwater visual intelligent perception and relates to a domain-adaptive marine benthic organism detection method based on consistent light field and color distribution. Background Art

[0002] In the task of detecting marine benthic organisms, the data samples used to train the deep convolutional neural network model and the test data samples often come from different water depths, different lighting conditions, and different sea environments, which directly leads to domain shift between the training set and the test set. [1] In this case, the detector can only achieve high-precision detection on a test set that has the same or similar domain distribution as the training set. [2] , it is difficult to achieve superior detection performance on test sets with different domain distributions [3] Therefore, it is urgent to propose a domain-adaptive marine benthic organism detection method.

[0003] To address the domain shift challenge in the detection of marine benthic organisms, existing methods mainly include: sample synthesis [4] , stage learning [5] and image enhancement [6] et al. With the help of YOLOv3 benchmark model, the literature [4] A water quality migration method was proposed to migrate the water type of the marine benthic organisms dataset, and a domain-invariant module was constructed by combining the feature extractor and domain classifier to achieve dataset augmentation. [7] A conditional bidirectional style transfer network was proposed, which used a linear transformation matrix to convert underwater images from a source domain to another domain, thereby increasing the number of training set samples; a novel domain mixing module was constructed to perform interpolation operations on two different domains at the feature level, enriching the domain diversity of the training dataset and improving the domain generalization performance of marine benthic organism detection.

[0004] By simulating the learning process of the human brain, the literature [5] By dividing the marine benthic samples into subsets based on resolution, and arranging the learning order of the subsets according to the classification and detection losses achieved by a general object detector on these subsets, a phased learning strategy, from simple to difficult, was proposed. Furthermore, a combination of multiple image processing methods was used to simultaneously enhance the sample images in the training and test sets, significantly narrowing the domain distribution differences between the two.

[0005] literature [6] By combining CLAHE, sharpening, morphology and brightness adjustment techniques, the contrast of underwater images is improved, the edge details of underwater images are enhanced, and the brightness of underwater images is improved, which reduces the impact of domain offset on the detection task of marine benthic organisms to a certain extent. [8]Instead of directly using degraded image samples as training sets, the Haar wavelet transform technique was used to transfer underwater image styles, effectively improving the generalization performance of the marine benthic organism detection network in different waters. [9] The multi-scale Retinex technology is used to enhance the images passed into the cross-domain collaborative network model. A gated feature fusion module is designed to adjust the mixing ratio of degraded images and enhanced images, making full use of the complementary information between the two and improving the detection performance in domain shift scenarios.

[0006] Through the review and analysis of existing work, the following deficiencies still exist in the fuzzy marine benthic organism detection:

[0007] (1) A review of existing research shows that the detection of marine benthic organisms in domain offset scenarios still has the following deficiencies: underwater images rendered using water mass migration methods and conditional bidirectional style transfer networks still lack authenticity to a certain extent, and deviate greatly from the actual domain distribution of underwater images.

[0008] (2) The staged learning method not only requires the construction of a general target detector in advance, but also requires the manual arrangement of the training order of different subsets based on the classification loss and detection loss, which makes the training steps too cumbersome.

[0009] (3) Although combining multiple traditional image enhancement algorithms can alleviate the domain shift phenomenon, the underwater light field and color distribution characteristics are not comprehensively considered in the loss function design process. Simply changing the pixel-level features cannot completely improve the domain distribution quality. Summary of the Invention

[0010] In order to solve the above problems, the technical solution adopted by the present invention is: a domain-adaptive marine benthic organism detection method based on consistent light field and color distribution, comprising the following steps:

[0011] Obtain paired images of degraded and reference marine benthic organisms containing any one or more of sea urchins, scallops, starfish, and sea cucumbers to construct a dataset;

[0012] Divide the data set into training set and test set;

[0013] Build a domain adaptive detection model based on light field and color distribution consistency;

[0014] The domain adaptive detection model based on consistent light field and color distribution is trained based on the training set data to obtain a trained domain adaptive detection model based on consistent light field and color distribution;

[0015] The test set data is input into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

[0016] Furthermore: the domain adaptive detection model based on consistent light field and color distribution includes:

[0017] The encoder-decoder domain converter module with a multi-residual mechanism performs feature extraction on the input paired degraded image and reference image to obtain the feature-extracted generated image;

[0018] Backbone network: performs feature extraction on the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I;

[0019] The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II;

[0020] The second upsampling module is configured to obtain a feature extraction image III by stitching the feature extraction images output by the intermediate component modules of the backbone network;

[0021] Detection head network: Achieve detection and identification of any one or more marine benthic targets such as large-scale, medium-scale, and small-scale sea urchins, scallops, starfish, and sea cucumbers;

[0022] Loss network: Based on the feature extraction image I output by the backbone network, the paired degraded image and reference image, the feature extraction image II, and the feature extraction image III, the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network are jointly optimized.

[0023] Furthermore: the loss network includes: a first loss network, a second loss network, and a third loss network;

[0024] The first loss network adopts a feature-aware loss module: for extracting an image I based on the features of the input, and improving the similarity between the generated image and the reference image from the feature level;

[0025] The second loss network includes: an underwater light field perception loss module, which is constructed by combining multi-scale Gaussian convolution, averaging and normalization to promote light field consistency between the generated image and the reference image;

[0026] The color distribution consistency loss module uses the convex hull to represent the feature scatter distribution of the generated image and the reference image in Lab color space. It is designed by considering the diagonal length of the minimum enclosing rectangle and the Euclidean distance between the convex hull center points to promote the color distribution consistency between the generated image and the reference image.

[0027] A low-frequency detail loss module improves the similarity between the generated image and the reference image at the pixel level; the underwater light field perception loss module, the color distribution consistency loss module, and the low-frequency detail loss module are respectively based on the input paired degraded image and the reference image to collaboratively improve the similarity between the generated image and the reference image;

[0028] The third loss network adopts a multi-scale detection loss module: based on the feature extraction image III combined with positioning loss, confidence loss and classification loss, to improve the accuracy of marine benthic organism detection.

[0029] Furthermore: the encoding-decoding domain converter module with a multi-residual mechanism includes an encoder and a decoder:

[0030] The encoder is constructed by extracting features through cascaded convolution, batch normalization and Leaky ReLU modules. To reduce the number of parameters, convolution with a kernel size of 3×3 and a stride of 2 is used to perform downsampling operations, and the resolution of the feature map is scaled by 1 / 2.

[0031] The decoder uses nearest neighbor interpolation between features and convolution with a kernel size of 3×3 and a stride of 1.

[0032] Furthermore, the construction process of the underwater light field perception loss module is as follows:

[0033] The underwater light field is extracted by constructing Gaussian weights, which is expressed as:

[0034]

[0035] Among them, W σ (i,j) is the Gaussian weight of position (i,j), represents the weight of the kth layer in the lth convolution kernel, σ∈{15,60,90} represents the Gaussian kernel width, i=1,L,w g ,j=1,L,h g ,l=1,L,n f and k=1,L,n c ;

[0036] Then, the convolution operation is used to extract the rough underwater light field, which is expressed as:

[0037]

[0038] Among them, S img ∈{ref,gen}, symbol represents convolution, and Represents the k-th layer features in the reference image and the generated image, is the rough feature of the first layer in the underwater light field image;

[0039] To remove the Information on mesobenthic organisms and a more granular underwater light field map Expressed as:

[0040]

[0041] in, lg(·) represents the logarithmic transformation function, and ξ represents a constant;

[0042] Furthermore, the average underwater light field map Expressed as:

[0043]

[0044] in,

[0045] Normalization techniques are used to reduce the influence of outliers, expressed as:

[0046]

[0047] Among them, max(·) and min(·) represent the maximum and minimum value operations respectively;

[0048] A more natural underwater light field map is represented as:

[0049]

[0050] in, and Represent the more natural reference image light field map and generated image light field map respectively;

[0051] Finally, the underwater light field perception loss l ULFP Expressed as:

[0052]

[0053] Furthermore: the construction process of the color distribution consistency loss module is as follows:

[0054] The reference images and generate images Convert from RGB color space to Lab color space using the convex hull C constructed from the polygon ref and C gen Represents the distribution of scattered points in Lab color space, convex hull C ref and C gen The coordinates of the polygon vertices are given by the set and Describe;

[0055] Correspondingly, the polygon convex hull C ref and C gen The area is expressed as:

[0056]

[0057] Among them, |·| represents the absolute value operation, s ref and s gen Represent the convex hull C ref and C gen area;

[0058] On the convex hull of the polygon C ref and C gen When there is an intersection between them, the intersection area s IARG Expressed as:

[0059]

[0060] Among them, Represents the convex hull C ref and C gen The coordinate set of the polygon vertices in the intersection area,

[0061] When the convex hull C ref and C gen When they intersect, the intersection ratio between them is expressed as:

[0062]

[0063] In order to further accurately describe the convex hull C ref and C gen The deviation between the two, taking into account the surrounding C ref and C gen The Euclidean distance between the diagonal length of the minimum enclosing rectangle and the corresponding center point is expressed as:

[0064]

[0065] Among them, d diag represents the diagonal length, d ctr represents the Euclidean distance between the center points of the convex hull,

[0066] Finally, the color distribution loss l CDC Expressed as:

[0067]

[0068] in, and are the reference image and the generated underwater image respectively.

[0069] Furthermore, the output of the feature perception loss module is expressed as:

[0070]

[0071] Here, φ(·) represents the VGG-19 feature extractor, c, h, and w represent the number of feature layers, the height, and the width of the feature map, respectively.

[0072] Furthermore: the multi-scale detection loss module includes a regression loss module, a confidence loss module, and a classification loss module. The output of the regression loss module is expressed as:

[0073]

[0074] in, represents the jth anchor box in the i-th grid responsible for detecting the current target, and are the center coordinates of the predicted box and the real box, and Represents the width and height of the predicted box and the real box, n k and n ab Represents the number of horizontal and vertical grids and the number of anchor boxes in a single grid;

[0075] In addition, the confidence loss module l conf The output is represented as:

[0076]

[0077] in, represents the jth anchor box in the i-th grid that is not responsible for detecting the current target, and are the predicted and true confidences, respectively;

[0078] In addition, the output of the classification loss module is expressed as:

[0079]

[0080] in, and are the predicted and true categories, respectively;

[0081] In this case, the multi-scale detection loss l MDL Expressed as:

[0082] l MDL =l reg +l conf +l cls (19)

[0083] Combined with the above formula, the entire optimization function can be expressed as:

[0084] l t =l ULFPL +l CDCL +l PEL +l FPL +l MDL (20)

[0085] A domain-adaptive marine benthic organism detection device based on consistent light field and color distribution, comprising:

[0086] Acquisition module: used to acquire paired degraded and reference marine benthic images containing any one or more of sea urchins, scallops, starfish, and sea cucumbers, and construct a dataset;

[0087] Partitioning module: used to divide the data set into training set and test set;

[0088] Building module: used to build a domain adaptive detection model based on light field and color distribution consistency;

[0089] Training module: used to train the domain adaptive detection model based on consistent light field and color distribution based on the training set data, and obtain a trained domain adaptive detection model based on consistent light field and color distribution;

[0090] Implementation module: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

[0091] The present invention provides a domain-adaptive marine benthic organism detection method based on consistent light field and color distribution, which has the following advantages:

[0092] Compared with the prior art, the present invention has the following beneficial effects:

[0093] 1. Generally speaking, underwater image information can be decomposed into the illumination component of natural light and the reflection component of the target object. Utilizing multi-scale Gaussian convolution technology, the natural light field information of the underwater scene is preserved, and the averaging operation generates a complementary underwater light field map.

[0094] 2. The developed underwater light field perception loss module can minimize the introduction of marine benthic biological structure information, accelerating the light field conversion process between the generated image and the reference image;

[0095] 3. In the Lab color space, especially when the convex hull of the generated image and the convex hull of the reference image do not intersect, the proposed color distribution consistency loss can still measure the distribution difference, promote the color distribution conversion between the generated image and the reference image, and thus alleviate the impact of domain shift on high-precision marine benthic organism detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0097] Figure 1 It is a marine benthic organism detection method based on LFCDC;

[0098] Figure 2 is the encoding-decoding domain converter module;

[0099] Figure 3 is the color distribution of the reference image and the generated image in Lab space;

[0100] Figure 4 Comparison diagram of underwater light field perception; (a) is the blue-green situation, (b) is the blue-biased situation, (c) is the green-biased situation, (d) is the bright light situation, (e) is the multi-target coexistence situation, and (f) is the low-light situation;

[0101] Figure 5 Qualitative comparison of domain conversion performance on the DUO dataset; (a) blue-green case, (b) blue-biased case, (c) green-biased case, (d) low-light case;

[0102] Figure 6 Color distribution of the degraded image, generated image, and reference image in Lab space; (a) blue-green case, (b) blue-biased case, (c) green-biased case;

[0103] Figure 7 Performance comparison of the accuracy-recall curves of different categories; (a) accuracy-recall curve of sea urchins, (b) accuracy-recall curve of scallops, (c) accuracy-recall curve of starfish, and (d) accuracy-recall curve of sea cucumbers;

[0104] Figure 8 Qualitative comparison of detection performance in typical scenes: (a) dark light, (b) bright light, (c) greenish, (d) bluish. DETAILED DESCRIPTION

[0105] It should be noted that, unless there is any conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0106] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0107] Figure 1 It is a marine benthic organism detection method based on LFCDC;

[0108] A domain-adaptive marine benthic organism detection method based on consistent light field and color distribution includes the following steps:

[0109] S1: Obtain paired degraded-reference marine benthic images and construct a dataset;

[0110] S2: Divide the data set into training set and test set;

[0111] S3: Build a domain adaptive detection model based on consistent light field and color distribution;

[0112] S4: training a domain adaptive detection model based on light field and color distribution consistency (LFCDC) based on the training set data to obtain a trained domain adaptive detection model based on light field and color distribution consistency;

[0113] S5: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to achieve detection of marine benthic organisms based on consistent light field and color distribution.

[0114] Steps S1 / S2 / S3 / S4 / S5A are performed sequentially;

[0115] In order to improve the detection performance of marine benthic organisms in the domain offset scenario, a domain adaptive detection model based on Light Field and Color Distribution Consistency (LFCDC) is proposed. Figure 1As shown in the figure, it mainly includes the Encoder-Decoder Domain Converter (EDDC) module, Underwater Light Field Perception Loss (ULFPL) and Color Distribution Consistency Loss (CDCL), and is trained using the degraded-reference marine benthic samples constructed by SARSDN.

[0116] The domain adaptive detection model based on consistent light field and color distribution includes:

[0117] Encoder-Decoder Domain Converter (EDDC) with multi-residual mechanism: It extracts features from the input paired degraded image and reference image to obtain the generated image after feature extraction.

[0118] Backbone network: performs feature extraction on the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I;

[0119] The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II;

[0120] The second upsampling module is configured to obtain a feature extraction image III by stitching the feature extraction images output by the intermediate component modules of the backbone network;

[0121] Loss network: Jointly optimizes the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network based on the feature extraction image I output by the backbone network, the paired degraded image and reference image, the feature extraction image II, and the feature extraction image III;

[0122] Detection head network: realizes the detection and identification of large-scale, medium-scale and small-scale marine benthic targets;

[0123] The backbone network includes a Res1 module, a Res2 module, a first Res8 module, a second Res8 module and a Res4 module;

[0124] The Res1 module, the Res2 module, the first Res8 module, the second Res8 module and the Res4 module are connected in sequence;

[0125] The output of the first Res8 module is further spliced ​​and connected with the second upsampling module to output a feature extraction image III;

[0126] The loss network includes: a first loss network, a second loss network, and a third loss network;

[0127] The first loss network adopts a feature-aware loss module: for extracting an image I based on the features of the input, and improving the similarity between the generated image and the reference image from the feature level;

[0128] The second loss network includes: an underwater light field perception loss module (ULFPL), which is constructed by combining multi-scale Gaussian convolution, averaging and normalization;

[0129] Color Distribution Consistency Loss (CDCL) is a module that uses the convex hull to represent the feature scatter distribution of the generated image and the reference image in Lab color space. It is designed by considering the diagonal length of the minimum enclosing rectangle and the Euclidean distance between the convex hull center points to promote light field consistency between the generated image and the reference image.

[0130] A low-frequency detail corruption module that improves the similarity between the generated image and the reference image at the pixel level;

[0131] The underwater light field perception loss module, the color distribution consistency loss module, and the low-frequency detail damage module work together to improve the similarity between the generated image and the reference image;

[0132] The third loss network adopts a multi-scale detection loss module: based on the feature extraction image III combined with positioning loss, confidence loss and classification loss, to improve the accuracy of marine benthic organism detection;

[0133] The encoder-decoder domain converter converts samples from multiple domains into a single domain, such as Figure 2 As shown in Figure 2, a UNet-like network with multiple residual mechanisms is used to construct the encoding-decoding domain converter module.

[0134] The encoding-decoding domain converter module with multi-residual mechanism includes an encoder and a decoder:

[0135] The encoder is constructed by extracting features through cascaded convolution, batch normalization and Leaky ReLU modules. To reduce the number of parameters, convolution with a kernel size of 3×3 and a stride of 2 is used to perform downsampling operations, and the resolution of the feature map is scaled by 1 / 2.

[0136] The decoder, in order to avoid checkerboard artifacts, uses nearest neighbor interpolation, convolution with kernel size of 3×3 and stride of 1.

[0137] The construction process of the underwater light field perception loss module is as follows:

[0138] The underwater light field is extracted by constructing Gaussian weights, which is expressed as:

[0139]

[0140] Among them, W σ (i,j) is the Gaussian weight of position (i,j), represents the weight of the kth layer in the lth convolution kernel, σ∈{15,60,90} represents the Gaussian kernel width, i=1,L,w g ,j=1,L,h g ,l=1,L,n f and k=1,L,n c ;

[0141] Then, the convolution operation is used to extract the rough underwater light field, which is expressed as:

[0142]

[0143] Among them, S img ∈{ref,gen}, symbol represents convolution, and Represents the k-th layer features in the reference image and the generated image, is the rough feature of the first layer in the underwater light field image;

[0144] To remove the Information on mesobenthic organisms and a more granular underwater light field map Expressed as:

[0145]

[0146] in, lg(·) represents the logarithmic transformation function, and ξ represents a constant;

[0147] Furthermore, the average underwater light field map Expressed as:

[0148]

[0149] in,

[0150] Normalization techniques are used to reduce the influence of outliers, expressed as:

[0151]

[0152] Among them, max(·) and min(·) represent the maximum and minimum value operations respectively;

[0153] A more natural underwater light field map is represented as:

[0154]

[0155] in, and Represent the more natural reference image light field map and generated image light field map respectively;

[0156] Finally, the underwater light field perception loss l ULFP Expressed as:

[0157]

[0158] Figure 3 is the color distribution of the reference image and the generated image in Lab space;

[0159] The construction process of the color distribution consistency loss module is as follows:

[0160] The reference images and generate images Convert from RGB color space to Lab color space using the convex hull C constructed from the polygon ref and C gen Represents the distribution of scattered points in Lab color space, convex hull C ref and C gen The coordinates of the polygon vertices are given by the set and Describe;

[0161] Correspondingly, the polygon convex hull C ref and C gen The area is expressed as:

[0162]

[0163] Among them, |·| represents the absolute value operation, s ref and s gen Represent the convex hull C ref and C gen area;

[0164] On the convex hull of the polygon C ref and C gen When there is an intersection between them, the intersection area s IARG Expressed as:

[0165]

[0166] Among them, Represents the convex hull C ref and C genThe coordinate set of the polygon vertices in the intersection area,

[0167] When the convex hull C ref and C gen When they intersect, the intersection ratio between them is expressed as:

[0168]

[0169] In order to further accurately describe the convex hull C ref and C gen The deviation between the two, taking into account the surrounding C ref and C gen The Euclidean distance between the diagonal length of the minimum enclosing rectangle and the corresponding center point is expressed as:

[0170]

[0171] Among them, d diag represents the diagonal length, d ctr represents the Euclidean distance between the center points of the convex hull,

[0172] Finally, the color distribution loss l CDC Expressed as:

[0173]

[0174]

[0175] in, and are the reference image and the generated underwater image respectively.

[0176] The output of the feature-aware loss module is expressed as:

[0177]

[0178] Here, φ(·) represents the VGG-19 feature extractor, c, h, and w represent the number of feature layers, the height, and the width of the feature map, respectively.

[0179] The multi-scale detection loss module includes a regression loss module, a confidence loss module and a classification loss module. The output of the regression loss module is expressed as:

[0180]

[0181] in, represents the jth anchor box in the i-th grid responsible for detecting the current target, and are the center coordinates of the predicted box and the real box, and Represents the width and height of the predicted box and the real box, n k and n ab Represents the number of horizontal and vertical grids and the number of anchor boxes in a single grid;

[0182] In addition, the confidence loss module l conf The output is represented as:

[0183]

[0184] in, represents the jth anchor box in the i-th grid that is not responsible for detecting the current target, and are the predicted and true confidences, respectively;

[0185] In addition, the output of the classification loss module is expressed as:

[0186]

[0187] in, and are the predicted and true categories, respectively;

[0188] In this case, the multi-scale detection loss l MDL Expressed as:

[0189] l MDL =l reg +l conf +l cls (19)

[0190] Combined with the above formula, the entire optimization function can be expressed as:

[0191] l t =l ULFPL +l CDCL +l PEL +l FPL +l MDL (20)

[0192] A domain-adaptive marine benthic organism detection device based on consistent light field and color distribution, comprising:

[0193] Acquisition module: used to acquire paired degraded and reference marine benthic images containing any one or more of sea urchins, scallops, starfish, and sea cucumbers, and construct a dataset;

[0194] Partitioning module: used to divide the data set into training set and test set;

[0195] Building module: used to build a domain adaptive detection model based on light field and color distribution consistency;

[0196] Training module: used to train the domain adaptive detection model based on consistent light field and color distribution based on the training set data, and obtain a trained domain adaptive detection model based on consistent light field and color distribution;

[0197] Implementation module: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

[0198] Example 1:

[0199] 1. Dataset and experimental configuration of the detection framework;

[0200] This paper conducts comprehensive comparative experiments on the URPC2020 dataset and the DUO dataset, which contain 5543 and 7782 images, respectively. For the URPC2020 dataset, the training set contains 4434 images and the test set contains 1109 images; for the DUO dataset, the training set contains 6671 images and the test set contains 1111 images. The batch size, image resolution, number of training rounds, initial learning rate, final learning rate, and constant ξ are set to 2, 512×512, 50, and 10, respectively. -3 , 10 -4 and 10 -5 To improve convergence performance, weights pre-trained on the COCO dataset were used. Furthermore, the present invention uses AP, mAP@.5, mAP@.75, and mAP@[.5,.95] as detection performance evaluation metrics. The present invention uses a five-fold cross-validation strategy, taking the average of the five test results as the detection accuracy of different models, and uses a video with a resolution of 1280×720 to obtain the model's inference speed.

[0201] 2. Comparison of underwater light field perception

[0202] In order to verify the underwater light field perception performance, this paper fully considers the blue-green, blue-biased, green-biased, bright light, multi-target coexistence and low-light scenes, and the corresponding extraction results are presented in Figure 4 Comparison diagram of underwater light field perception; (a) is the blue-green situation, (b) is the blue-biased situation, (c) is the green-biased situation, (d) is the bright light situation, (e) is the multi-target coexistence situation, and (f) is the low-light situation;

[0203] Overall, the designed underwater light field perception module can construct a very detailed underwater light field, and mainly focuses on the light field information of the underwater scene, avoiding the introduction of the structure and texture information of the foreground objects to the greatest extent.

[0204] 3. Qualitative comparison of domain conversion effects

[0205] In order to intuitively verify the superiority of the LFCDC method in domain conversion, the present invention considers the physical model-based restoration method (IBLA), the model-free enhancement method (RGHS) and the underwater image enhancement method based on deep learning (FUnIE-GAN, UGAN, UWCNN and Zero-DCE), and the corresponding experiments are carried out in cyan, blue-biased, green-biased and low-light scenes. Figure 5 Qualitative comparison of domain conversion performance on the DUO dataset: (a) cyan, (b) bluish, (c) greenish, and (d) low-light conditions. As shown, the IBLA and RGHS methods fail to completely remove color casts. This is primarily due to the difficulty of accurately estimating the intermediate parameters involved in physical model-based restoration methods, while model-free enhancement methods fail to consider the degradation mechanism of degraded images and therefore have poor generalization. Furthermore, the UWCNN method, which directly learns the nonlinear mapping between the degraded and reference images, also fails to completely remove color casts, demonstrating its extreme dependence on the quality and quantity of training samples. Although both FUnIE-GAN and UGAN utilize generative adversarial networks, their enhancement results are significantly weaker than the proposed LFCDC method. This suggests that simultaneously considering both underwater light field perception loss and color distribution consistency loss can make the enhanced images more consistent with human visual perception. Note that the Zero-DCE method, which directly estimates the zero-reference depth curve, only improves image brightness but fails to effectively suppress the color cast in degraded images, exacerbating the "fogging" effect of underwater images to some extent. Overall, the LFCDC method achieves the most superior domain transfer performance in the considered scenarios.

[0206] 4. Quantitative comparison of domain conversion effects

[0207] Furthermore, a quantitative comparison of domain conversion performance on the DUO dataset shows, as shown in Table 1, that the LFCDC method significantly outperforms other methods except for the UISM metric. The proposed LFCDC method achieves the best UIQM and UCIQE metric values, indicating that the enhanced images achieved by the LFCDC method achieve an optimal balance in color, clarity, and contrast, and that the enhanced underwater images are more consistent with the human visual perception system.

[0208] Table 1 Quantitative comparison of domain conversion performance on the DUO dataset

[0209]

[0210] 5. Comparison of color distribution in Lab space

[0211] Figure 6Color distribution of the degraded image, generated image, and reference image in Lab space; (a) blue-green case, (b) blue-biased case, (c) green-biased case;

[0212] To facilitate visualization, Figure 6 The color distribution of the degraded image, generated image, and reference image in Lab space is presented, and the scenes compared include blue-green, blue-biased, and green-biased. Figure 6 As can be seen in the Lab color space, the characteristic scatter points of the degraded image (i.e., orange patches) are far from the origin and appear as elongated strips. Furthermore, the characteristic scatter points of the generated image (i.e., cyan patches) are evenly distributed around the origin, with their a and b values ​​falling within the intervals of (-20, 20) and (-20, 40), respectively. Clearly, the generated image obtained using the LFCDC method has almost identical color distribution characteristics to the reference image. Therefore, underwater images of varying degradation types can be converted to the same domain distribution, effectively mitigating the impact of domain shift on marine benthic biodetection.

[0213] 6. Module ablation experiment in LFCDC method

[0214] To fully demonstrate the contributions of the designed EDDC, ULFPL, and CDCL modules to the LFCDC method, we selected YOLOv3 as the baseline model. As shown in Table 2, the baseline model's detection accuracy is significantly weaker than that of other methods combining EDDC, ULFPL, or CDCL modules. The proposed LFCDC method achieves the best mAP@.5, mAP@.75, and mAP@[.5,.95] metrics, demonstrating that the LFCDC method effectively suppresses domain shift. Furthermore, combining the baseline model with the EDDC module can transform degraded image samples to the same domain, thus achieving superior detection performance compared to the baseline model. Removing the ULFPL module results in a decrease in detection accuracy, as measured by the mAP@.5, mAP@.75, and mAP@[.5,.95] metrics, demonstrating that the ULFPL module maintains consistency in the underwater light field between the generated and reference images. Furthermore, removing the CDCL module also results in a degradation in detection accuracy, indicating that the difference in color distribution between the training and test sets significantly impacts detection accuracy.

[0215] Table 2 Ablation experiments of different modules on the DUO dataset

[0216]

[0217] Furthermore, Figure 7Precision-recall curves for sea urchins, scallops, starfish, and sea cucumbers are shown: (a) sea urchin, (b) scallop, (c) starfish, and (d) sea cucumber. It can be seen that, compared to the baseline model, combining the EDDC, ULFPL, or CDCL modules yields more competitive detection performance. In particular, for sea urchins, starfish, and sea cucumbers, the LFCDC method achieves the optimal balance between precision and recall.

[0218] 7. Quantitative comparison of specialized marine benthic organism detection methods

[0219] To verify the superiority of the LFCDC method, we conducted quantitative comparisons with specialized marine benthic organism detection methods, including the Collaborative Framework for Underwater Object Detection (CFUOD), TVFRD, LUOD, CSPTCenterNet, BBROABR, FCSA, and DDRL. The corresponding results are shown in Table 3. As can be seen, the LUOD method, using MobileNet v2 as its backbone network, achieved the fastest detection speed (44 FPS) at a 416×416 resolution. By designing an autocorrelation total variation loss and a feature reconstruction loss, the CFUOD method suppresses domain shift by correcting color and improving contrast, achieving the best performance in detecting scallops. The CSPTCenterNet method uses a Transformer architecture to enhance the encoder's global information extraction capabilities and utilizes the GIoU technique to improve the accuracy of marine benthic organism bounding box regression. Because the TVFRD method requires a region proposal network to generate a large number of candidate bounding boxes, its inference speed only reaches 3 FPS. Furthermore, while BBROABR, FCSA, and DDRL methods improve marine benthic organism detection accuracy by improving bounding box regression localization performance, recalibrating relevant feature map weights, and transferring head class embedding feature distribution information, respectively, failing to account for domain shift can lead to severe degradation in detection performance. By combining an encoder-decoder domain converter module, an underwater light field perception loss module, and a color distribution consistency loss module, and supported by paired degraded-reference marine benthic organism samples, the LFCDC method effectively suppresses the impact of domain shift, achieving the highest mAP@.5, mAP@.75, and mAP@[.5,.95] metrics.

[0220] Table 3 Quantitative comparison of dedicated marine benthic organism detection methods on the DUO dataset

[0221]

[0222] 8. Quantitative comparison of common target detection methods

[0223] Furthermore, in order to demonstrate the superiority of the LFCDC method, the present invention comprehensively considers the two-stage detection method based on anchor boxes (Faster R-CNN), the single-stage detection method based on anchor boxes (SSD, EfficientDet, YOLOv4, YOLOv7, RFBNet and RetinaNet) and the detection method based on no anchor boxes (FCOS and CenterNet), and the corresponding experiments are carried out on the URPC2020 dataset. As shown in Table 4, the detection accuracy of FCOS and CenterNet is lower than that of the proposed LFCDC method. The main reason is that the anchor box information is not utilized during the training process. Although the construction of a robust feature extractor can obtain high-level semantic information, the detection accuracy of EfficientDet, YOLOv4 and YOLOv7 is still lower than that of the proposed LFCDC method. The key reason is that the domain shift problem between the training set and the test set is not considered. By focusing on difficult samples and reducing the weight of simple instance samples, RetinaNet can only improve the detection accuracy to a limited extent by constructing a focal loss function. Since two-stage detection methods require the use of a region proposal network to generate a large number of candidate bounding boxes, Faster R-CNN struggles to achieve real-time detection. The proposed LFCDC method achieves the most competitive detection performance, based on mAP@.5, mAP@.75, and mAP@[.5,.95] metrics. By combining the EDDC, ULFPL, and CDCL modules, the proposed method achieves the most competitive detection performance.

[0224] Table 4 Quantitative comparison of general target detection methods on URPC2020 dataset

[0225]

[0226] 9. Qualitative comparison of marine benthic organism detection

[0227] Finally, to demonstrate the superiority of the LFCDC method in real underwater environments, this paper considers two-stage detection methods (Faster R-CNN), single-stage detection methods (SSD, EfficientDet, YOLOv4 and YOLOv7) and anchor-free detection methods (CenterNet), and the corresponding experiments are carried out in dark light scenes, bright light scenes, green scenes and blue scenes. Figure 8Qualitative comparison of detection performance in typical scenarios: (a) dim light, (b) bright light, (c) greenish, and (d) bluish. In dim light, Faster R-CNN achieves high-precision detection and recognition thanks to the Region Proposal Network (RPN) generating a large number of high-quality candidate bounding boxes. YOLOv7 also achieves excellent detection performance thanks to its extended efficient layer aggregation network and hierarchical label assignment strategy. In bright light, SSD, YOLOv4, and CenterNet experience a large number of missed detections. In greenish and bluish scenes, YOLOv7 and LFCDC both achieve superior detection performance. Furthermore, SSD, EfficientDet, and CenterNet fail to detect all sea urchins. The key reason is that they fail to account for the domain shift between the training set and the actual test scene. Combining the EDDC, ULFPL, and CDCL modules, the LFCDC method ensures consistent light field and color distribution between the training and test sets, thereby suppressing domain shift.

[0228] 10. Quantitative comparison of domain adaptation detection methods

[0229] To fully demonstrate the superiority of the designed LFCDC method, the present invention considers classic domain adaptation detection methods, including DG-YOLO, DMCL, EnYOLO, RoIMix, and UWYOLOX. Specifically, as shown in Table 5, although the EnYOLO algorithm and the UWYOLOX algorithm respectively propose a domain adaptation mechanism based on multi-scale training and a pseudo-label learning strategy based on a teacher model, they struggle to achieve detection accuracy consistent with the LFCDC method. Note that the aforementioned algorithms can achieve superior detection speed, mainly due to the use of a lightweight detection framework. In addition, although the DG-YOLO algorithm uses a domain classifier to facilitate the network to mine domain-invariant semantic feature information, the detection accuracy of different categories is still lower than that of the proposed LFCDC framework. It is worth noting that for the sea urchin and starfish categories, the DMCL method achieves the highest detection accuracy, mainly because the combined use of conditional bidirectional style transfer technology, domain mixing strategy, and spatial selectivity margin contrast loss can increase domain diversity and normalize features. Overall, the proposed DMCL framework is able to achieve the best detection accuracy in terms of mAP@.5 and mAP@[.5,.95] metrics.

[0230] Table 5 Performance comparison of domain adaptive detection methods

[0231]

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain-adaptive marine benthic organism detection method based on consistent light field and color distribution, characterized by: The following steps are involved: Obtain paired images of degraded and reference marine benthic organisms containing any one or more of sea urchins, scallops, starfish, and sea cucumbers to construct a dataset; Divide the data set into training set and test set; Build a domain adaptive detection model based on light field and color distribution consistency; The domain adaptive detection model based on consistent light field and color distribution is trained based on the training set data to obtain a trained domain adaptive detection model based on consistent light field and color distribution; The domain adaptive detection model based on consistent light field and color distribution includes: The encoder-decoder domain converter module with a multi-residual mechanism performs feature extraction on the input paired degraded image and reference image to obtain the feature-extracted generated image; Backbone network: performs feature extraction on the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I; The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II; The second upsampling module is configured to obtain a feature extraction image III by stitching the feature extraction images output by the intermediate component modules of the backbone network; Detection head network: Achieve detection and identification of any one or more marine benthic targets such as large-scale, medium-scale, and small-scale sea urchins, scallops, starfish, and sea cucumbers; Loss network: Jointly optimizes the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network based on the feature extraction image I output by the backbone network, the paired degraded image and reference image, the feature extraction image II, and the feature extraction image III; The loss network includes: a first loss network and a second loss network; The first loss network adopts a feature-aware loss module: for extracting an image I based on the features of the input, and improving the similarity between the generated image and the reference image from the feature level; The second loss network includes: an underwater light field perception loss module, which is constructed by combining multi-scale Gaussian convolution, averaging and normalization to promote light field consistency between the generated image and the reference image; The color distribution consistency loss module uses the convex hull to represent the feature scatter distribution of the generated image and the reference image in Lab color space. It is designed by considering the diagonal length of the minimum enclosing rectangle and the Euclidean distance between the convex hull center points to promote the color distribution consistency between the generated image and the reference image. The output of the feature-aware loss module is expressed as: (15) in, represents the VGG-19 feature extractor, , and Respectively represent the number of feature layers, the height and width of the feature map; The test set data is input into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

2. The domain-adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 1, characterized in that: The loss network includes: a third loss network; Low-frequency detail corruption module; improves the similarity between the generated image and the reference image at the pixel level; The underwater light field perception loss module, the color distribution consistency loss module and the low-frequency detail damage module are respectively based on the input paired degraded image and the reference image to collaboratively improve the similarity between the generated image and the reference image; The third loss network adopts a multi-scale detection loss module: based on the feature extraction image III combined with positioning loss, confidence loss and classification loss, to improve the accuracy of marine benthic organism detection.

3. The domain-adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 1, characterized in that: The encoding-decoding domain converter module with multi-residual mechanism includes an encoder and a decoder: The encoder is constructed by extracting features through cascaded convolution, batch normalization and Leaky ReLU modules. To reduce the number of parameters, convolution with a kernel size of 3×3 and a stride of 2 is used to perform downsampling operations, and the resolution of the feature map is scaled by 1 / 2. The decoder uses nearest neighbor interpolation between features and convolution with a kernel size of 3×3 and a stride of 1.

4. The domain-adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 2, characterized in that: The construction process of the underwater light field perception loss module is as follows: The underwater light field is extracted by constructing Gaussian weights, which is expressed as: (1) in, It's location Gaussian weights, Representative The convolution kernel layer weights, represents the Gaussian kernel width, , , and ; Then, the convolution operation is used to extract the rough underwater light field, which is expressed as: (2) in, ,symbol" " represents convolution, and Represents the reference image and the generated image Layer features, The underwater light field image Layer roughness characteristics; To remove the Information on mesobenthic organisms and a more granular underwater light field map Expressed as: (3) in, , represents the logarithmic transformation function, represents a constant value; Furthermore, the average underwater light field map Expressed as: (4) in, ; Normalization techniques are used to reduce the influence of outliers, expressed as: (5) in, and Represents the maximum and minimum operations respectively; A more natural underwater light field map is represented as: (6) in, and Represent the more natural reference image light field map and generated image light field map respectively; Ultimately, underwater light field perception is lost Expressed as: (7)。 5. The domain-adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 1, characterized in that: The construction process of the color distribution consistency loss module is as follows: The reference images and generate images Convert from RGB color space to Lab color space using the convex hull constructed from polygons and Represents the distribution of scattered points in Lab color space, convex hull and The coordinates of the polygon vertices are given by the set and Describe; Correspondingly, the convex hull of the polygon and The area is expressed as: (8) Among them, 、 、 、 , Represents the absolute value operation, and Represents the convex hull and area; On the convex hull of the polygon and When there is an intersection between them, the intersection area Expressed as: (9) Among them, , , Represents the convex hull and The coordinate set of the polygon vertices in the intersection area, ; When the convex hull and When they intersect, the intersection ratio between them is expressed as: (10) In order to further accurately describe the convex hull and The deviation between and The Euclidean distance between the diagonal length of the minimum enclosing rectangle and the corresponding center point is expressed as: (11) (12) in, represents the diagonal length, , represents the Euclidean distance between the center points of the convex hull, ; Finally, the color distribution is uniformly lost Expressed as: (13) (14) in, and are the reference image and the generated underwater image respectively.

6. The domain-adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 2, characterized in that: The multi-scale detection loss module includes a regression loss module, a confidence loss module, and a classification loss module. The output of the regression loss module is expressed as: (16) in, Representative In the grid An anchor box responsible for detecting the current target, and are the center coordinates of the predicted box and the real box, and Represents the width and height of the predicted box and the real box, and Represents the number of horizontal and vertical grids and the number of anchor boxes in a single grid; In addition, the confidence loss module The output is represented as: (17) in, Representative In the grid An anchor box that is not responsible for detecting the current target, and are the predicted and true confidence, respectively; In addition, the output of the classification loss module is expressed as: (18) in, and are the predicted and true categories, respectively; In this case, the multi-scale detection loss Expressed as: (19) Combined with the above formula, the entire optimization function can be expressed as: （20）。 7. A domain-adaptive marine benthic organism detection device based on consistent light field and color distribution, characterized by: include: Acquisition module: used to acquire paired degraded and reference marine benthic images containing any one or more of sea urchins, scallops, starfish, and sea cucumbers, and construct a dataset; Partitioning module: used to divide the data set into training set and test set; Building module: used to build a domain adaptive detection model based on light field and color distribution consistency; Training module: used to train the domain adaptive detection model based on consistent light field and color distribution based on the training set data, and obtain a trained domain adaptive detection model based on consistent light field and color distribution; The domain adaptive detection model based on consistent light field and color distribution includes: The encoder-decoder domain converter module with a multi-residual mechanism performs feature extraction on the input paired degraded image and reference image to obtain the feature-extracted generated image; Backbone network: performs feature extraction on the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I; The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II; The second upsampling module is configured to obtain a feature extraction image III by stitching the feature extraction images output by the intermediate component modules of the backbone network; Detection head network: Achieve detection and identification of any one or more marine benthic targets such as large-scale, medium-scale, and small-scale sea urchins, scallops, starfish, and sea cucumbers; Loss network: Jointly optimizes the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network based on the feature extraction image I output by the backbone network, the paired degraded image and reference image, the feature extraction image II, and the feature extraction image III; The loss network includes: a first loss network and a second loss network; The first loss network adopts a feature-aware loss module: for extracting an image I based on the features of the input, and improving the similarity between the generated image and the reference image from the feature level; The second loss network includes: an underwater light field perception loss module, which is constructed by combining multi-scale Gaussian convolution, averaging and normalization to promote light field consistency between the generated image and the reference image; The color distribution consistency loss module uses the convex hull to represent the feature scatter distribution of the generated image and the reference image in Lab color space. It is designed by considering the diagonal length of the minimum enclosing rectangle and the Euclidean distance between the convex hull center points to promote the color distribution consistency between the generated image and the reference image. The output of the feature-aware loss module is expressed as: (15) in, represents the VGG-19 feature extractor, , and Respectively represent the number of feature layers, the height and width of the feature map; Implementation module: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

Images