Domain self-adaptive marine benthos detection method based on light field and color distribution consistency

Abstract

The invention relates to a domain self-adaptive marine benthos detection method based on light field and color distribution consistency, which comprises the following steps of: obtaining a pair of degradation-reference marine benthos images with any one or more of sea urchins, scallops, starfishes and sea cucumbers, and constructing a data set; dividing the data set to obtain a training set and a test set; constructing a domain adaptive detection model based on light field and color distribution consistency; training the domain adaptive detection model based on the light field and color distribution consistency based on the training set data to obtain a trained domain adaptive detection model based on the light field and color distribution consistency; and inputting the data of the test set into the trained domain adaptive detection model based on the consistent light field and color distribution to realize detection of any one or more of sea urchins, scallops, starfishes and sea cucumbers based on the consistent light field and color distribution.

Description

Technical Field

[0001] The invention belongs to the field of underwater visual intelligent perception and relates to a domain adaptive marine benthic organism detection method based on consistent light field and color distribution. Background Art

[0002] In the task of detecting benthic organisms in the ocean, the data samples used to train the deep convolutional neural network model and the test data samples often come from different water depths, different lighting conditions, and different sea environments, which directly leads to domain shift between the training set and the test set. [1] In this case, the detector can only achieve high-precision detection on a test set that has the same or similar domain distribution as the training set. [2] , it is difficult to achieve superior detection performance on test sets with different domain distributions [3] Therefore, it is urgent to propose a domain-adaptive marine benthic organism detection method.

[0003] To address the challenge of domain shift in the detection of marine benthic organisms, existing methods mainly include: sample synthesis [4] , stage learning [5] and image enhancement [6] et al. With the help of YOLOv3 benchmark model, the literature [4] A water quality migration method was proposed to migrate the water type of the marine benthic organisms dataset, and a domain-invariant module was constructed by combining the feature extractor and domain classifier to achieve dataset augmentation. [7] A conditional bidirectional style transfer network was proposed, which used a linear transformation matrix to convert underwater images from a source domain to another domain, thereby increasing the number of samples in the training set. A novel domain mixing module was constructed to perform interpolation operations on two different domains at the feature level, enriching the domain diversity of the training dataset and improving the domain generalization performance of marine benthic organism detection.

[0004] By simulating the learning process of the human brain, [5] The samples of benthic organisms in the ocean are divided into different subsets according to the resolution, and the learning order of the subsets is arranged according to the classification loss and detection loss achieved by the general object detector on different subsets, thus proposing a phased learning strategy from simple to difficult. In addition, a variety of image processing methods are combined to simultaneously enhance the sample images of the training set and the test set, greatly reducing the difference in domain distribution between the two.

[0005] literature [6] By combining CLAHE, sharpening, morphology and brightness adjustment techniques, the underwater image contrast is improved, the edge details of the underwater image are enhanced, and the underwater image brightness is improved, which reduces the impact of domain offset on the marine benthic organism detection task to a certain extent. [8]Instead of directly using degraded image samples as training sets, the Haar wavelet transform technique was used to transfer the underwater image style, which effectively improved the generalization performance of the marine benthic organism detection network in different waters. [9] The multi-scale Retinex technology is used to enhance the images passed into the cross-domain collaborative network model, and a gated feature fusion module is designed to adjust the mixing ratio of degraded images and enhanced images, making full use of the complementary information between the two and improving the detection performance in domain shift scenarios.

[0006] Through the review and analysis of existing work, the following deficiencies still exist in the fuzzy marine benthic organism detection:

[0007] (1) A review of existing research shows that the detection of marine benthic organisms in domain offset scenarios still has the following shortcomings: the underwater images rendered using the water mass migration method and the conditional bidirectional style transfer network still lack authenticity to a certain extent, and have a large deviation from the actual underwater image domain distribution.

[0008] (2) The phased learning method not only requires the construction of a general target detector in advance, but also requires the manual arrangement of the training order of different subsets based on the classification loss and detection loss, which makes the training steps too cumbersome.

[0009] (3) Although combining multiple traditional image enhancement algorithms can alleviate the domain shift phenomenon, the underwater light field and color distribution characteristics are not comprehensively considered in the process of loss function design. Simply changing the pixel-level features cannot completely improve the domain distribution quality. Summary of the invention

[0010] In order to solve the above problems, the technical solution adopted by the present invention is: a domain adaptive marine benthic organism detection method based on consistent light field and color distribution, comprising the following steps:

[0011] Obtain paired images of degraded and reference marine benthic organisms with any one or more of sea urchins, scallops, starfish, and sea cucumbers to construct a data set;

[0012] Divide the data set into training set and test set;

[0013] Construct a domain adaptive detection model based on light field and color distribution consistency;

[0014] Based on the training set data, a domain adaptive detection model based on consistent light field and color distribution is trained to obtain a trained domain adaptive detection model based on consistent light field and color distribution;

[0015] The test set data is input into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

[0016] Further: the domain adaptive detection model based on consistent light field and color distribution includes:

[0017] The encoding-decoding domain converter module with multi-residual mechanism extracts features from the input paired degraded image and reference image to obtain the generated image after feature extraction.

[0018] Backbone network: extracting features from the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I;

[0019] The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II;

[0020] The second upsampling module: splicing the feature extraction images output by the intermediate component modules of the backbone network to obtain the feature extraction image III;

[0021] Detection head network: Realizes the detection and recognition of any one or more marine benthic biological targets such as large-scale, medium-scale and small-scale sea urchins, scallops, starfish and sea cucumbers;

[0022] Loss network: Based on the feature extraction image I, paired degraded images and reference images, feature extraction image II, and feature extraction image III output by the backbone network, the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network are jointly optimized.

[0023] Further: the loss network includes: a first loss network, a second loss network, and a third loss network;

[0024] The first loss network adopts a feature-aware loss module: used to extract the image I based on the input features, and improve the similarity between the generated image and the reference image from the feature level;

[0025] The second loss network includes: an underwater light field perception loss module, which is constructed by combining multi-scale Gaussian convolution, averaging and normalization to promote the light field consistency of the generated image and the reference image;

[0026] The color distribution consistency loss module is based on the convex hull to represent the feature scatter distribution of the generated image and the reference image in the Lab color space. It is designed by considering the diagonal length of the minimum closed rectangle and the Euclidean distance between the center points of the convex hull to promote the color distribution consistency between the generated image and the reference image.

[0027] A low-frequency detail damage module; improving the similarity between the generated image and the reference image from a pixel level; the underwater light field perception loss module, the color distribution consistency loss module and the low-frequency detail damage module are respectively based on the input paired degraded image and the reference image, and collaboratively improve the similarity between the generated image and the reference image;

[0028] The third loss network adopts a multi-scale detection loss module: based on the feature extraction image III combined with positioning loss, confidence loss and classification loss, the accuracy of marine benthic organism detection is improved.

[0029] Further: the encoding-decoding domain converter module with a multi-residual mechanism includes an encoder and a decoder:

[0030] The encoder is constructed to extract features through cascaded convolution, batch normalization and Leaky ReLU modules. To reduce the number of parameters, a convolution with a kernel size of 3×3 and a step size of 2 is used to perform downsampling operations, and the feature map resolution is scaled by 1 / 2 in turn;

[0031] The decoder uses the nearest neighbor interpolation between features and convolution with a kernel size of 3×3 and a stride of 1.

[0032] Further: the construction process of the underwater light field perception loss module is as follows:

[0033] The underwater light field is extracted by constructing Gaussian weights, which is expressed as:

[0034]

[0035] Among them, W σ (i,j) is the Gaussian weight of position (i,j), represents the weight of the kth layer in the lth convolution kernel, σ∈{15,60,90} represents the Gaussian kernel width, i=1,L,w g ,j=1,L,h g ,l=1,L,n f and k=1,L,n c ;

[0036] Then, the convolution operation is used to extract the rough underwater light field, which is expressed as:

[0037]

[0038] Among them, S img ∈{ref,gen}, symbol represents convolution, and Represents the k-th layer features in the reference image and the generated image, is the rough feature of the lth layer in the underwater light field image;

[0039] To remove the Information on mesobenthic organisms, more granular underwater light field maps It is expressed as:

[0040]

[0041] in, lg(·) represents the logarithmic transformation function, and ξ represents a constant;

[0042] Furthermore, the average underwater light field map It is expressed as:

[0043]

[0044] in,

[0045] Use normalization techniques to reduce the impact of outliers, expressed as:

[0046]

[0047] Among them, max(·) and min(·) represent the maximum and minimum value operations respectively;

[0048] A more natural underwater light field map is represented as:

[0049]

[0050] in, and They represent more natural reference image light field maps and generated image light field maps respectively;

[0051] Finally, the underwater light field perception loss l ULFP It is expressed as:

[0052]

[0053] Further: the construction process of the color distribution consistency loss module is as follows:

[0054] The reference images and generate images Convert from RGB color space to Lab color space using the convex hull C constructed from the polygon ref and C gen Represents the distribution of scattered points in Lab color space, convex hull C ref and C gen The coordinates of the polygon vertices are given by the set and Give a description;

[0055] Correspondingly, the convex hull C of the polygon ref and C gen The area is expressed as:

[0056]

[0057] Among them, |·| represents the absolute value operation, s ref and gen Represent the convex hull C ref and C gen area;

[0058] On the convex hull of the polygon C ref and C gen If there is an intersection between them, the intersection area s IARG It is expressed as:

[0059]

[0060] Among them, Represents the convex hull C ref and C gen The coordinate set of the polygon vertices in the intersection area.

[0061] When the convex hull C ref and C gen When they intersect, the intersection ratio between them is expressed as:

[0062]

[0063] In order to further accurately describe the convex hull C ref and C gen The deviation between the two takes into account the surrounding C ref and C gen The Euclidean distance between the diagonal length of the minimum enclosing rectangle and the corresponding center point is expressed as:

[0064]

[0065] Among them, d diag represents the length of the diagonal line, d ctr represents the Euclidean distance between the center points of the convex hull,

[0066] Finally, the color distribution loss l CDC It is expressed as:

[0067]

[0068] in, and They are the reference image and the generated underwater image respectively.

[0069] Further: the output of the feature-aware loss module is expressed as:

[0070]

[0071] Here, φ(·) represents the VGG-19 feature extractor, c, h, and w represent the number of feature layers, the height and width of the feature map, respectively.

[0072] Further: The multi-scale detection loss module includes a regression loss module, a confidence loss module and a classification loss module. The output of the regression loss module is expressed as:

[0073]

[0074] in, represents the jth anchor box in the i-th grid responsible for detecting the current target, and are the center coordinates of the predicted box and the true box, and Represents the width and height of the predicted box and the real box, n k and n ab Represents the number of horizontal and vertical grids and the number of anchor boxes in a single grid;

[0075] In addition, the confidence loss module l conf The output is represented as:

[0076]

[0077] in, represents the jth anchor box in the i-th grid that is not responsible for detecting the current target. and are the predicted and true confidences, respectively;

[0078] In addition, the output of the classification loss module is expressed as:

[0079]

[0080] in, and are the predicted and true categories, respectively;

[0081] In this case, the multi-scale detection loss l MDL It is expressed as:

[0082] l MDL = l reg +l conf +l cls (19)

[0083] Combined with the above formula, the entire optimization function can be expressed as:

[0084] l t = l ULFPL +l CDCL +l PEL +l FPL +l MDL (20)

[0085] A domain-adaptive marine benthic organism detection device based on consistent light field and color distribution, comprising:

[0086] Acquisition module: used to acquire paired images of degraded and reference marine benthic organisms with any one or more of sea urchins, scallops, starfish, and sea cucumbers, and construct a data set;

[0087] Partitioning module: used to divide the data set into training set and test set;

[0088] Building module: used to build a domain adaptive detection model based on light field and color distribution consistency;

[0089] Training module: used to train the domain adaptive detection model based on consistent light field and color distribution based on the training set data, so as to obtain a trained domain adaptive detection model based on consistent light field and color distribution;

[0090] Implementation module: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

[0091] The present invention provides a domain-adaptive marine benthic organism detection method based on consistent light field and color distribution, which has the following advantages:

[0092] Compared with the prior art, the present invention has the following beneficial effects:

[0093] 1. Generally speaking, underwater image information can be decomposed into the illumination component of natural light and the reflection component of the target object. The multi-scale Gaussian convolution technique is used to retain the natural light field information of the underwater scene, and the averaging operation generates a complementary underwater light field map.

[0094] 2. The developed underwater light field perception loss module can avoid introducing the structure information of marine benthic organisms to the greatest extent, and accelerate the light field conversion process between the generated image and the reference image;

[0095] 3. In the Lab color space, especially when there is no intersection between the convex hull of the generated image and the convex hull of the reference image, the proposed color distribution consistency loss can still measure the distribution difference, promote the color distribution conversion between the generated image and the reference image, and thus reduce the impact of domain offset on high-precision marine benthic organism detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0097] Figure 1 It is a marine benthic organism detection method based on LFCDC;

[0098] Figure 2 is the encoding-decoding domain converter module;

[0099] Figure 3 is the color distribution of the reference image and the generated image in Lab space;

[0100] Figure 4 Comparison diagram of underwater light field perception; (a) is the blue-green situation, (b) is the blue-biased situation, (c) is the green-biased situation, (d) is the bright light situation, (e) is the multi-target coexistence situation, and (f) is the low light situation;

[0101] Figure 5 Qualitative comparison of domain conversion performance on the DUO dataset; (a) blue-green case, (b) blue-biased case, (c) green-biased case, (d) low-light case;

[0102] Figure 6 Color distribution of the degraded image, generated image, and reference image in Lab space; (a) blue-green case, (b) blue-biased case, (c) green-biased case;

[0103] Figure 7 Performance comparison of accuracy-recall curves of different categories; (a) accuracy-recall curve of sea urchins, (b) accuracy-recall curve of scallops, (c) accuracy-recall curve of starfish, (d) accuracy-recall curve of sea cucumbers;

[0104] Figure 8 Qualitative comparison of detection performance in typical scenes: (a) dark light, (b) bright light, (c) greenish, and (d) bluish. DETAILED DESCRIPTION

[0105] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0106] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0107] Figure 1 It is a marine benthic organism detection method based on LFCDC;

[0108] A domain adaptive marine benthic organism detection method based on consistent light field and color distribution includes the following steps:

[0109] S1: Obtain paired degraded-reference marine benthic images and construct a dataset;

[0110] S2: Divide the data set into training set and test set;

[0111] S3: Build a domain adaptive detection model based on consistent light field and color distribution;

[0112] S4: training a domain adaptive detection model based on light field and color distribution consistency (LFCDC) based on the training set data to obtain a trained domain adaptive detection model based on light field and color distribution consistency;

[0113] S5: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of marine benthic organisms based on consistent light field and color distribution.

[0114] The steps S1 / S2 / S3 / S4 / S5A are performed sequentially;

[0115] In order to improve the detection performance of marine benthic organisms in the domain offset scenario, a domain adaptive detection model based on light field and color distribution consistency (LFCDC) is proposed. Figure 1As shown in the figure, it mainly includes the Encoder-Decoder Domain Converter (EDDC) module, the Underwater Light Field Perception Loss (ULFPL) and the Color Distribution Consistency Loss (CDCL), and the training is completed using the degraded-reference marine benthic samples constructed by SARSDN;

[0116] The domain adaptive detection model based on consistent light field and color distribution includes:

[0117] Encoder-Decoder Domain Converter module with multi-residual mechanism, EDDC: extracts features from the input paired degraded image and reference image to obtain the generated image after feature extraction;

[0118] Backbone network: extracting features from the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I;

[0119] The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II;

[0120] The second upsampling module: splicing the feature extraction images output by the intermediate component modules of the backbone network to obtain the feature extraction image III;

[0121] Loss network: based on the feature extraction image I output by the backbone network, the paired degraded image and reference image, the feature extraction image II, and the feature extraction image III, jointly optimize the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network;

[0122] Detection head network: realizes the detection and identification of large-scale, medium-scale and small-scale marine benthic biological targets;

[0123] The backbone network includes a Res1 module, a Res2 module, a first Res8 module, a second Res8 module and a Res4 module;

[0124] The Res1 module, the Res2 module, the first Res8 module, the second Res8 module and the Res4 module are connected in sequence;

[0125] The output of the first Res8 module is also spliced ​​and connected with the second upsampling module to output a feature extraction image III;

[0126] The loss network includes: a first loss network, a second loss network, and a third loss network;

[0127] The first loss network adopts a feature-aware loss module: used to extract the image I based on the input features, and improve the similarity between the generated image and the reference image from the feature level;

[0128] The second loss network includes: an underwater light field perception loss module (Underwater Light Field Perception Loss, ULFPL), which is constructed by combining multi-scale Gaussian convolution, averaging and normalization;

[0129] Color Distribution Consistency Loss (CDCL), based on the convex hull representation of the feature scatter distribution of the generated image and the reference image in the Lab color space, is designed by considering the diagonal length of the minimum closed rectangle and the Euclidean distance between the center points of the convex hull to promote the light field consistency between the generated image and the reference image;

[0130] A low-frequency detail corruption module that improves the similarity between the generated image and the reference image at the pixel level;

[0131] The underwater light field perception loss module, the color distribution consistency loss module, and the low-frequency detail damage module work together to improve the similarity between the generated image and the reference image;

[0132] The third loss network adopts a multi-scale detection loss module: based on the feature extraction image III combined with positioning loss, confidence loss and classification loss, the accuracy of marine benthic organism detection is improved;

[0133] The encoding-decoding domain converter converts samples from multiple domains into a single domain, such as Figure 2 As shown, a UNet-like network with multiple residual mechanisms is used to construct the encoding-decoding domain converter module.

[0134] The encoding-decoding domain converter module with multi-residual mechanism includes an encoder and a decoder:

[0135] The encoder is constructed to extract features through cascaded convolution, batch normalization and Leaky ReLU modules. To reduce the number of parameters, a convolution with a kernel size of 3×3 and a step size of 2 is used to perform downsampling operations, and the feature map resolution is scaled by 1 / 2 in turn;

[0136] The decoder, in order to avoid checkerboard artifacts, uses nearest neighbor interpolation, convolution with a kernel size of 3×3 and a stride of 1.

[0137] The construction process of the underwater light field perception loss module is as follows:

[0138] The underwater light field is extracted by constructing Gaussian weights, which is expressed as:

[0139]

[0140] Among them, W σ (i,j) is the Gaussian weight of position (i,j), represents the weight of the kth layer in the lth convolution kernel, σ∈{15,60,90} represents the Gaussian kernel width, i=1,L,w g ,j=1,L,h g ,l=1,L,n f and k=1,L,n c ;

[0141] Then, the convolution operation is used to extract the rough underwater light field, which is expressed as:

[0142]

[0143] Among them, S img ∈{ref,gen}, symbol represents convolution, and Represents the k-th layer features in the reference image and the generated image, is the rough feature of the lth layer in the underwater light field image;

[0144] To remove the Information on mesobenthic organisms, more granular underwater light field maps It is expressed as:

[0145]

[0146] in, lg(·) represents the logarithmic transformation function, and ξ represents a constant;

[0147] Furthermore, the average underwater light field map It is expressed as:

[0148]

[0149] in,

[0150] Use normalization techniques to reduce the impact of outliers, expressed as:

[0151]

[0152] Among them, max(·) and min(·) represent the maximum and minimum value operations respectively;

[0153] A more natural underwater light field map is represented as:

[0154]

[0155] in, and They represent more natural reference image light field maps and generated image light field maps respectively;

[0156] Finally, the underwater light field perception loss l ULFP It is expressed as:

[0157]

[0158] Figure 3 is the color distribution of the reference image and the generated image in Lab space;

[0159] The construction process of the color distribution consistency loss module is as follows:

[0160] The reference images and generate images Convert from RGB color space to Lab color space using the convex hull C constructed from the polygon ref and C gen Represents the distribution of scattered points in Lab color space, convex hull C ref and C gen The coordinates of the polygon vertices are given by the set and Give a description;

[0161] Correspondingly, the convex hull C of the polygon ref and C gen The area is expressed as:

[0162]

[0163] Among them, |·| represents the absolute value operation, s ref and gen Represent the convex hull C ref and C gen area;

[0164] On the convex hull of the polygon C ref and C gen If there is an intersection between them, the intersection area s IARG It is expressed as:

[0165]

[0166] Among them, Represents the convex hull C ref and C genThe coordinate set of the polygon vertices in the intersection area.

[0167] When the convex hull C ref and C gen When they intersect, the intersection ratio between them is expressed as:

[0168]

[0169] In order to further accurately describe the convex hull C ref and C gen The deviation between the two takes into account the surrounding C ref and C gen The Euclidean distance between the diagonal length of the minimum enclosing rectangle and the corresponding center point is expressed as:

[0170]

[0171] Among them, d diag represents the length of the diagonal line, d ctr represents the Euclidean distance between the center points of the convex hull,

[0172] Finally, the color distribution loss l CDC It is expressed as:

[0173]

[0174]

[0175] in, and They are the reference image and the generated underwater image respectively.

[0176] The output of the feature-aware loss module is expressed as:

[0177]

[0178] Here, φ(·) represents the VGG-19 feature extractor, c, h, and w represent the number of feature layers, the height and width of the feature map, respectively.

[0179] The multi-scale detection loss module includes a regression loss module, a confidence loss module and a classification loss module. The output of the regression loss module is expressed as:

[0180]

[0181] in, represents the jth anchor box in the i-th grid responsible for detecting the current target, and are the center coordinates of the predicted box and the true box, and Represents the width and height of the predicted box and the real box, n k and n ab Represents the number of horizontal and vertical grids and the number of anchor boxes in a single grid;

[0182] In addition, the confidence loss module l conf The output is represented as:

[0183]

[0184] in, represents the jth anchor box in the i-th grid that is not responsible for detecting the current target. and are the predicted and true confidences, respectively;

[0185] In addition, the output of the classification loss module is expressed as:

[0186]

[0187] in, and are the predicted and true categories, respectively;

[0188] In this case, the multi-scale detection loss l MDL It is expressed as:

[0189] l MDL = l reg +l conf +l cls (19)

[0190] Combined with the above formula, the entire optimization function can be expressed as:

[0191] l t = l ULFPL +l CDCL +l PEL +l FPL +l MDL (20)

[0192] A domain-adaptive marine benthic organism detection device based on consistent light field and color distribution, comprising:

[0193] Acquisition module: used to acquire paired images of degraded and reference marine benthic organisms with any one or more of sea urchins, scallops, starfish, and sea cucumbers, and construct a data set;

[0194] Partitioning module: used to divide the data set into training set and test set;

[0195] Building module: used to build a domain adaptive detection model based on light field and color distribution consistency;

[0196] Training module: used to train the domain adaptive detection model based on consistent light field and color distribution based on the training set data, so as to obtain a trained domain adaptive detection model based on consistent light field and color distribution;

[0197] Implementation module: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

[0198] Embodiment 1:

[0199] 1. Dataset and experimental configuration of the detection framework;

[0200] This paper conducts comprehensive comparative experiments on the URPC2020 dataset and the DUO dataset, which contain 5543 images and 7782 images respectively. For the URPC2020 dataset, the training set contains 4434 images and the test set contains 1109 images; for the DUO dataset, the training set contains 6671 images and the test set contains 1111 images. The batch size, image resolution, number of training rounds, initial learning rate, end learning rate and constant ξ are set to 2, 512×512, 50, 10, respectively. -3 , 10 -4 and 10 -5 . In order to improve the convergence performance, the weights pre-trained on the COCO dataset are used. In addition, the present invention uses AP, mAP@.5, mAP@.75 and mAP@[.5,.95] as detection performance evaluation indicators. The present invention uses a five-fold cross-validation strategy, takes the average of the five test results as the detection accuracy of different models, and uses a video with a resolution of 1280×720 to obtain the inference speed of the model.

[0201] 2. Comparison of underwater light field perception

[0202] In order to verify the underwater light field perception performance, the present invention fully considers the blue-green, blue-biased, green-biased, bright light, multi-target coexistence and low-light scenes, and the corresponding extraction results are presented in Figure 4 Comparison diagram of underwater light field perception; (a) is the blue-green situation, (b) is the blue-biased situation, (c) is the green-biased situation, (d) is the bright light situation, (e) is the multi-target coexistence situation, and (f) is the low light situation;

[0203] Overall, the designed underwater light field perception module can construct a very detailed underwater light field, and mainly focuses on the light field information of the underwater scene, avoiding the introduction of the structure and texture information of the foreground objects to the greatest extent.

[0204] 3. Qualitative comparison of domain conversion effects

[0205] In order to intuitively verify the superiority of the LFCDC method in domain conversion, the present invention considers the restoration method based on physical model (IBLA), the model-free enhancement method (RGHS) and the underwater image enhancement method based on deep learning (FUnIE-GAN, UGAN, UWCNN and Zero-DCE), and the corresponding experiments are carried out in cyan, blue-biased, green-biased and low-light scenes. Figure 5 Qualitative comparison of domain conversion performance on the DUO dataset; (a) blue-green case, (b) blue-biased case, (c) green-biased case, (d) low-light case; As shown, the IBLA and RGHS methods cannot completely remove the color cast. The main reason is that the restoration method based on the physical model is difficult to accurately estimate the intermediate parameters involved, and the model-free enhancement method does not consider the degradation mechanism of the degraded image and has poor generalization. In addition, by directly learning the nonlinear mapping relationship between the degraded image and the reference image, the UWCNN method also cannot completely remove the color cast, indicating that this method is extremely dependent on the quality and quantity of training samples. Although both the FUnIE-GAN and UGAN methods use the generative adversarial network mechanism, the enhancement effect is significantly weaker than the proposed LFCDC method, which shows that considering the underwater light field perception loss and the color distribution consistency loss at the same time can make the enhanced image more consistent with human visual perception characteristics. Note that the Zero-DCE method that directly estimates the zero-reference depth curve can only improve the image brightness, but cannot effectively suppress the color cast of the degraded image, which to some extent aggravates the "fogging" effect of the underwater image. Overall, the LFCDC method can achieve the most superior domain transfer performance in the considered scenarios.

[0206] 4. Quantitative comparison of domain conversion effects

[0207] In addition, regarding the quantitative comparison of domain conversion performance on the DUO dataset, it can be seen from Table 1 that, except for the UISM indicator, the LFCDC method can significantly outperform other methods. The proposed LFCDC method can obtain the best UIQM and UCIQE indicator values, which means that the image enhanced by the LFCDC method can achieve the optimal balance in color, clarity and contrast, and the enhanced underwater image is more consistent with the human visual perception system.

[0208] Table 1 Quantitative comparison of domain conversion performance on DUO dataset

[0209]

[0210] 5. Color distribution comparison in Lab space

[0211] Figure 6Color distribution of the degraded image, generated image, and reference image in Lab space; (a) blue-green case, (b) blue-biased case, (c) green-biased case;

[0212] To facilitate visualization, Figure 6 The color distribution of the degraded image, generated image, and reference image in Lab space is presented, and the scenes compared include cyan, bluish, and greenish. Figure 6 It can be seen that in the Lab color space, the characteristic scatter points of the degraded image (i.e., orange patches) are far away from the origin and are distributed in thin strips. In addition, the characteristic scatter points of the generated image (i.e., cyan patches) are evenly distributed around the origin, and the a and b values ​​of the characteristic scatter points fall into the intervals of (-20, 20) and (-20, 40), respectively. Obviously, the generated image obtained by the LFCDC method has almost the same color distribution characteristics as the reference image, so underwater images of different degradation types can be converted into the same domain distribution, which effectively reduces the impact of domain offset on the detection of marine benthic organisms.

[0213] 6. Module ablation experiment in LFCDC method

[0214] In order to fully reveal the contribution of the designed EDDC, ULFPL and CDCL modules to the LFCDC method, the present invention selects YOLOv3 as the benchmark model. As shown in Table 2, the detection accuracy of the benchmark model is significantly weaker than that of other methods combined with EDDC, ULFPL or CDCL modules, and the proposed LFCDC method can obtain the most superior mAP@.5, mAP@.75 and mAP@[.5,.95] index values, which shows that the LFCDC method can effectively suppress the domain shift phenomenon. In addition, the combination of the benchmark model and the EDDC module can convert the degraded image samples to the same domain, thus achieving better detection performance than the benchmark model. According to the mAP@.5, mAP@.75 and mAP@[.5,.95] index values, removing the ULFPL module will lead to a decrease in detection accuracy, which shows that the ULFPL module can keep the underwater light field between the generated image and the reference image consistent. In addition, removing the CDCL module also leads to the degradation of detection accuracy, which shows that the color distribution difference between the training set and the test set greatly affects the detection accuracy.

[0215] Table 2 Ablation experiments of different modules on DUO dataset

[0216]

[0217] Further, Figure 7The accuracy-recall curves of sea urchins, scallops, starfishes, and sea cucumbers are given (a) sea urchin accuracy-recall curve, (b) scallop accuracy-recall curve, (c) starfish accuracy-recall curve, (d) sea cucumber accuracy-recall curve; it can be intuitively seen that compared with the baseline model, combining EDDC, ULFPL or CDCL modules can achieve more competitive detection performance. Especially for sea urchins, starfishes, and sea cucumbers, the LFCDC method can achieve the best balance between accuracy and recall.

[0218] 7. Quantitative comparison of special marine benthic organism detection methods

[0219] To verify the superiority of the LFCDC method, the present invention conducts quantitative comparison with dedicated marine benthic organism detection methods, including the Collaborative Framework for Underwater Object Detection (CFUOD), TVFRD, LUOD, CSPTCenterNet, BBROABR, FCSA and DDRL, and the corresponding results are shown in Table 3. It can be seen that the LUOD method uses MobileNet v2 as the backbone network and achieves the fastest detection speed (44FPS) at a resolution of 416×416. By designing the autocorrelation total variation loss and feature reconstruction loss, the CFUOD method suppresses domain shift from the perspective of correcting color and improving contrast, and obtains the best performance when detecting scallop categories. The CSPTCenterNet method uses the Transformer structure to enhance the global information extraction capability of the encoder, and uses the GIoU technology to improve the regression accuracy of the marine benthic organism bounding box. Since the TVFRD method needs to use the region recommendation network to generate a large number of candidate bounding boxes, the inference speed can only reach 3FPS. In addition, although the BBROABR, FCSA, and DDRL methods improve the accuracy of marine benthic organism detection from the perspectives of improving bounding box regression positioning performance, recalibrating relevant feature map weights, and migrating head class embedding feature distribution information, respectively, not considering the domain shift problem will cause serious degradation of detection performance. By combining the encoding-decoding domain converter module, the underwater light field perception loss module, and the color distribution consistency loss module, the LFCDC method effectively suppresses the influence of domain shift under the support of paired degraded-reference marine benthic organism samples, thereby achieving the highest mAP@.5, mAP@.75, and mAP@[.5,.95] indicator values.

[0220] Table 3 Quantitative comparison of dedicated marine benthic organism detection methods on the DUO dataset

[0221]

[0222] 8. Quantitative comparison of common target detection methods

[0223] Furthermore, in order to demonstrate the superiority of the LFCDC method, the present invention comprehensively considers two-stage detection methods based on anchor boxes (Faster R-CNN), single-stage detection methods based on anchor boxes (SSD, EfficientDet, YOLOv4, YOLOv7, RFBNet and RetinaNet) and detection methods based on no anchor boxes (FCOS and CenterNet), and the corresponding experiments are carried out on the URPC2020 dataset. As shown in Table 4, the detection accuracy of FCOS and CenterNet is lower than that of the proposed LFCDC method. The main reason is that the anchor box information is not used during the training process. Although the construction of a robust feature extractor can obtain high-level semantic information, the detection accuracy of EfficientDet, YOLOv4 and YOLOv7 is still lower than that of the proposed LFCDC method. The key reason is that the domain shift problem between the training set and the test set is not considered. By focusing on difficult samples and reducing the weight of simple instance samples, RetinaNet can only improve limited detection accuracy by constructing a focal loss function. Since the two-stage detection method needs to generate a large number of candidate bounding boxes using the region proposal network, it is difficult for Faster R-CNN to achieve real-time detection. According to the mAP@.5, mAP@.75 and mAP@[.5,.95] indicator values, the proposed LFCDC method can achieve the most competitive detection performance by combining the EDDC, ULFPL and CDCL modules.

[0224] Table 4 Quantitative comparison of general target detection methods on the URPC2020 dataset

[0225]

[0226] 9. Qualitative comparison of marine benthic organisms detection

[0227] Finally, to prove the superiority of LFCDC method in real underwater environment, this paper considers two-stage detection method (Faster R-CNN), single-stage detection method (SSD, EfficientDet, YOLOv4 and YOLOv7) and anchor-free detection method (CenterNet), and the corresponding experiments are carried out in dark light scene, bright light scene, green scene and blue scene. Figure 8Qualitative comparison of detection performance in typical scenes, (a) dark light, (b) bright light, (c) greenish, (d) blueish. In dark light scenes, Faster R-CNN achieves high-precision detection and recognition with the help of the region proposal network to generate a large number of high-quality candidate bounding boxes. YOLOv7 can also achieve excellent detection performance because it builds an extended efficient layer aggregation network and a hierarchical label allocation strategy. In bright light scenes, SSD, YOLOv4 and CenterNet have a large number of missed detections. In greenish and blueish scenes, YOLOv7 and LFCDC methods can achieve superior detection performance. In addition, SSD, EfficientDet and CenterNet cannot detect all sea urchins. The key reason is that they do not consider the domain shift problem between the training set and the actual test scene. Combined with the EDDC, ULFPL and CDCL modules, the LFCDC method makes the light field and color distribution in the training set and the test set consistent, thereby suppressing the domain shift phenomenon.

[0228] 10. Quantitative comparison of domain adaptation detection methods

[0229] In order to fully demonstrate the superiority of the designed LFCDC method, the present invention considers classic domain adaptive detection methods, including DG-YOLO, DMCL, EnYOLO, RoIMix and UWYOLOX methods. Specifically, as shown in Table 5, although the EnYOLO algorithm and the UWYOLOX algorithm respectively proposed a domain adaptive mechanism based on multi-scale training and a pseudo-label learning strategy based on a teacher model, it is difficult to achieve detection accuracy consistent with the LFCDC method. Note that the aforementioned algorithms can achieve a more superior detection speed, mainly because of the use of a lightweight detection framework. In addition, although the DG-YOLO algorithm uses a domain classifier to promote the network to mine domain-invariant semantic feature information, the detection accuracy of different categories is still lower than that of the proposed LFCDC framework. It should be emphasized that for the sea urchin and starfish categories, the DMCL method achieves the highest detection accuracy, mainly because the combined use of conditional bidirectional style transfer technology, domain mixing strategy and spatial selectivity margin contrast loss can increase domain diversity and normalized features. Overall, the proposed DMCL framework is able to achieve the best detection accuracy in terms of mAP@.5 and mAP@[.5,.95] metrics.

[0230] Table 5 Performance comparison of domain adaptive detection methods

[0231]

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain adaptive marine benthic organism detection method based on consistent light field and color distribution, characterized by: The following steps are involved: Obtain paired images of degraded and reference marine benthic organisms with any one or more of sea urchins, scallops, starfish, and sea cucumbers to construct a data set; Divide the data set into training set and test set; Construct a domain adaptive detection model based on light field and color distribution consistency; Based on the training set data, a domain adaptive detection model based on consistent light field and color distribution is trained to obtain a trained domain adaptive detection model based on consistent light field and color distribution; The test set data is input into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

2. According to claim 1, a domain adaptive marine benthic organism detection method based on consistent light field and color distribution is characterized by: The domain adaptive detection model based on consistent light field and color distribution includes: The encoding-decoding domain converter module with multi-residual mechanism extracts features from the input paired degraded image and reference image to obtain the generated image after feature extraction. Backbone network: extracting features from the generated image transmitted by the encoding-decoding domain converter module to obtain a feature-extracted image I; The first upsampling module: splicing the feature extraction image I output by the backbone network to obtain the feature extraction image II; The second upsampling module: splicing the feature extraction images output by the intermediate component modules of the backbone network to obtain the feature extraction image III; Detection head network: Realizes the detection and recognition of any one or more marine benthic biological targets such as large-scale, medium-scale and small-scale sea urchins, scallops, starfish and sea cucumbers; Loss network: Based on the feature extraction image I, paired degraded images and reference images, feature extraction image II, and feature extraction image III output by the backbone network, the trainable parameters contained in the encoding-decoding domain converter, the backbone network, and the detection head network are jointly optimized.

3. The domain adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 2, characterized in that: The loss network includes: a first loss network, a second loss network, and a third loss network; The first loss network adopts a feature-aware loss module: used to extract the image I based on the input features, and improve the similarity between the generated image and the reference image from the feature level; The second loss network includes: an underwater light field perception loss module, which is constructed by combining multi-scale Gaussian convolution, averaging and normalization to promote the light field consistency of the generated image and the reference image; The color distribution consistency loss module is based on the convex hull to represent the feature scatter distribution of the generated image and the reference image in the Lab color space. It is designed by considering the diagonal length of the minimum closed rectangle and the Euclidean distance between the center points of the convex hull to promote the color distribution consistency between the generated image and the reference image. Low-frequency detail corruption module; improves the similarity between the generated image and the reference image at the pixel level; The underwater light field perception loss module, the color distribution consistency loss module and the low-frequency detail damage module are respectively based on the input paired degraded image and the reference image to collaboratively improve the similarity between the generated image and the reference image; The third loss network adopts a multi-scale detection loss module: based on the feature extraction image III combined with positioning loss, confidence loss and classification loss, the accuracy of marine benthic organism detection is improved.

4. The domain adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 1, characterized in that: The encoding-decoding domain converter module with multi-residual mechanism includes an encoder and a decoder: The encoder is constructed to extract features through cascaded convolution, batch normalization and Leaky ReLU modules. To reduce the number of parameters, a convolution with a kernel size of 3×3 and a step size of 2 is used to perform downsampling operations, and the feature map resolution is scaled by 1 / 2 in turn; The decoder uses the nearest neighbor interpolation between features and convolution with a kernel size of 3×3 and a stride of 1.

5. The domain adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 3, characterized in that: The construction process of the underwater light field perception loss module is as follows: The underwater light field is extracted by constructing Gaussian weights, which is expressed as: Among them, W σ (i,j) is the Gaussian weight of position (i,j), represents the weight of the kth layer in the lth convolution kernel, σ∈{15,60,90} represents the Gaussian kernel width, i=1,L,w g ,j=1,L,h g ,l=1,L,n f and k=1,L,n c ; Then, the convolution operation is used to extract the rough underwater light field, which is expressed as: Among them, S img ∈{ref,gen}, symbol represents convolution, and Represents the k-th layer features in the reference image and the generated image, is the rough feature of the lth layer in the underwater light field image; To remove the Information on mesobenthic organisms, more granular underwater light field maps It is expressed as: in, lg(·) represents the logarithmic transformation function, and ξ represents a constant; Furthermore, the average underwater light field map It is expressed as: in, Use normalization techniques to reduce the impact of outliers, expressed as: Among them, max(·) and min(·) represent the maximum and minimum value operations respectively; A more natural underwater light field map is represented as: in, and They represent more natural reference image light field maps and generated image light field maps respectively; Finally, the underwater light field perception loss l ULFP It is expressed as:

6. The domain adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 3, characterized in that: The construction process of the color distribution consistency loss module is as follows: The reference images and generate images Convert from RGB color space to Lab color space using the convex hull C constructed from the polygon ref and C gen Represents the distribution of scattered points in Lab color space, convex hull C ref and C gen The coordinates of the polygon vertices are given by the set and Give a description; Correspondingly, the convex hull C of the polygon ref and C gen The area is expressed as: Among them, |·| represents the absolute value operation, s r ef and gen Represent the convex hull C r ef and C gen area; On the convex hull of the polygon C ref and C gen If there is an intersection between them, the intersection area s IARG It is expressed as: Among them, Represents the convex hull C r ef and C gen The coordinate set of the polygon vertices in the intersection area. When the convex hull C r ef and C gen When they intersect, the intersection ratio between them is expressed as: In order to further accurately describe the convex hull C r ef and C gen The deviation between the two takes into account the surrounding C r ef and C gen The Euclidean distance between the diagonal length of the minimum enclosing rectangle and the corresponding center point is expressed as: Among them, d diag represents the length of the diagonal line, d ctr represents the Euclidean distance between the center points of the convex hull, Finally, the color distribution loss l CDC It is expressed as: in, and They are the reference image and the generated underwater image respectively.

7. The domain adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 3, characterized in that: The output of the feature-aware loss module is expressed as: Here, φ(·) represents the VGG-19 feature extractor, c, h, and w represent the number of feature layers, the height and width of the feature map, respectively.

8. The domain adaptive marine benthic organism detection method based on consistent light field and color distribution according to claim 3, characterized in that: The multi-scale detection loss module includes a regression loss module, a confidence loss module, and a classification loss module. The output of the regression loss module is expressed as: in, represents the jth anchor box in the i-th grid responsible for detecting the current target, and are the center coordinates of the predicted box and the true box, and Represents the width and height of the predicted box and the real box, n k and n ab Represents the number of horizontal and vertical grids and the number of anchor boxes in a single grid; In addition, the confidence loss module l conf The output is represented as: in, represents the jth anchor box in the i-th grid that is not responsible for detecting the current target. and are the predicted and true confidences, respectively; In addition, the output of the classification loss module is expressed as: in, and are the predicted and true categories, respectively; In this case, the multi-scale detection loss l MDL It is expressed as: l MDL ＝l reg +l conf +l cls (19) Combined with the above formula, the entire optimization function can be expressed as: l t ＝l ULFPL +l CDCL +l PEL +l FPL +l MDL (20)。 9. A domain-adaptive marine benthic organism detection device based on consistent light field and color distribution, characterized in that: include: Acquisition module: used to acquire paired images of degraded and reference marine benthic organisms with any one or more of sea urchins, scallops, starfish, and sea cucumbers, and construct a data set; Partitioning module: used to divide the data set into training set and test set; Building module: used to build a domain adaptive detection model based on light field and color distribution consistency; Training module: used to train the domain adaptive detection model based on consistent light field and color distribution based on the training set data, so as to obtain a trained domain adaptive detection model based on consistent light field and color distribution; Implementation module: Input the test set data into the trained domain adaptive detection model based on consistent light field and color distribution to realize the detection of any one or more marine benthic organisms such as sea urchins, scallops, starfish, and sea cucumbers based on consistent light field and color distribution.

Images