A target segmentation method and device, electronic equipment and storage medium

Abstract

The application discloses a target segmentation method and device, electronic equipment and storage medium, and belongs to the technical field of image processing. In the method, for a target image collected underwater, a segmentation model performs target segmentation after image enhancement by an enhancement network. The enhancement network separates the target image on each color channel of RGB, extracts features of an initial feature map on each color channel after the separation, splices the extracted feature maps, obtains a feature map of the target image on multiple feature channels, and performs image enhancement on the target image based on the feature map of the target image on the multiple feature channels and weights of each feature channel. The weight determination rule includes that the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel. In this way, the feature information of the R color channel can be highlighted, the missing image details due to the lack of red light underwater can be filled, and the accuracy of subsequent target segmentation can be improved.

Description

A target segmentation method, apparatus, electronic device, and storage medium Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of marine ecology research, it is often necessary to segment underwater images to determine the precise outlines of marine organisms such as shells, corals, and fish.

[0003] Traditional target segmentation schemes rely on primary segmentation models such as YOLOv10-seg, but these are not optimized for the specific characteristics of the underwater environment, and their direct use has some drawbacks. For example, underwater target images suffer from red light loss and blue-green light retention due to light attenuation, and are also subject to severe noise interference. As a result, the feature extraction performance of the segmentation model is poor after inputting the image into the underwater image, leading to poor target segmentation results. Summary of the Invention

[0004] This application provides a target segmentation method, apparatus, electronic device, and storage medium to improve the target segmentation effect of target images acquired from underwater.

[0005] In a first aspect, embodiments of this application provide a target segmentation method, including:

[0006] The target image acquired underwater is input into the segmentation model to obtain the target segmentation result;

[0007] The segmentation model processes the target image according to the following steps:

[0008] Image enhancement is performed on the target image using an enhancement network;

[0009] Based on the enhanced target image, the target segmentation result is determined;

[0010] The enhancement network performs image enhancement according to the following steps:

[0011] The target image is separated into three color channels: red, green, and blue (RGB), to obtain an initial feature map for each color channel.

[0012] Feature extraction is performed on the initial feature map of each color channel to obtain feature maps on multiple feature channels corresponding to each color channel;

[0013] The feature maps of multiple feature channels corresponding to each color channel are stitched together to obtain the feature map of the target image in multiple feature channels.

[0014] Based on the feature maps of the target image in multiple feature channels and the weight of each feature channel, image enhancement is performed on the target image. The weight determination rule for each feature channel includes: the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channels corresponding to the G color channel and the B color channel.

[0015] Secondly, embodiments of this application provide a target segmentation apparatus, comprising:

[0016] The acquisition module is used to acquire target images collected from underwater.

[0017] The segmentation module is used to input the target image into the segmentation model to obtain the target segmentation result;

[0018] The segmentation model processes the target image according to the following steps:

[0019] Image enhancement is performed on the target image using an enhancement network;

[0020] Based on the enhanced target image, the target segmentation result is determined;

[0021] The enhancement network performs image enhancement according to the following steps:

[0022] The target image is separated into three color channels: red, green, and blue (RGB), to obtain an initial feature map for each color channel.

[0023] Feature extraction is performed on the initial feature map of each color channel to obtain feature maps on multiple feature channels corresponding to each color channel;

[0024] The feature maps of multiple feature channels corresponding to each color channel are stitched together to obtain the feature map of the target image in multiple feature channels.

[0025] Based on the feature maps of the target image in multiple feature channels and the weight of each feature channel, image enhancement is performed on the target image. The weight determination rule for each feature channel includes: the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channels corresponding to the G color channel and the B color channel.

[0026] Thirdly, embodiments of this application provide an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0027] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform any of the target segmentation methods described above.

[0028] Fourthly, embodiments of this application provide a storage medium in which, when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device is capable of executing any of the above-described target segmentation methods.

[0029] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described target segmentation methods.

[0030] In this embodiment, for a target image acquired underwater, the segmentation model first enhances the image using a built-in enhancement network, and then segments the target based on the enhanced image. The enhancement network's processing flow includes: separating the target image across the RGB color channels to obtain an initial feature map for each color channel; extracting features from the initial feature maps for each color channel to obtain feature maps for multiple feature channels corresponding to each color channel; concatenating the multi-channel feature maps for each color channel to obtain feature maps of the target image across multiple feature channels; and then enhancing the target image based on the feature maps of the target image across multiple feature channels and the weights of each feature channel. The weights of each feature channel include the weights of the feature channels corresponding to the R color channel, which are greater than the weights of the feature channels corresponding to the G and B color channels. In this way, extracting features from the initial feature maps of each color channel individually enhances the ability of each color channel to represent the features of the segmented target. Giving the R channel a higher weight effectively highlights red-light-related image features, compensating for details lost due to red light attenuation in the underwater environment, thereby enhancing the color balance and content expression of the target image. This provides a high-quality input image for subsequent target segmentation, thus improving segmentation accuracy. Furthermore, embedding the enhancement network into the segmentation model allows for joint training with the model, eliminating the need for additional image preprocessing outside the segmentation model. This end-to-end optimization significantly improves the adaptability and processing efficiency of the segmentation model for underwater images. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 is an application scenario diagram of a target segmentation method provided in an embodiment of this application;

[0033] Figure 2 is a schematic diagram of the structure of a YOLOv10-seg model provided in an embodiment of this application;

[0034] Figure 3 is a flowchart of ICAM processing of a target image according to an embodiment of this application;

[0035] Figure 4 is a schematic diagram of an ICAM structure provided in an embodiment of this application;

[0036] Figure 5 is a processing flowchart of a C2f-DPS provided in an embodiment of this application;

[0037] Figure 6 is a structural schematic diagram of a C2f-DPS provided in an embodiment of this application;

[0038] Figure 7 is a flowchart of a GMA processing embodiment provided in this application;

[0039] Figure 8 is a schematic diagram of a GMA structure provided in an embodiment of this application;

[0040] Figure 9 is a schematic diagram of a segmentation model provided in an embodiment of this application;

[0041] Figure 10 is a schematic diagram of a target segmentation device provided in an embodiment of this application;

[0042] Figure 11 is a schematic diagram of the hardware structure of an electronic device for implementing a target segmentation method according to an embodiment of this application. Detailed Implementation

[0043] To improve the target segmentation effect of target images acquired from underwater, embodiments of this application provide a target segmentation method, apparatus, electronic device, and storage medium.

[0044] Referring to Figure 1, Figure 1 is an application scenario diagram of a target segmentation method provided in an embodiment of this application. The ground processing device is located on the ground, and the underwater detection device is located underwater. The ground processing device and the underwater detection device can communicate with each other via the Internet.

[0045] Generally, underwater detection equipment is equipped with cameras to capture images of targets underwater. These images may include various marine life, such as fish, corals, turtles, seaweed, crabs, and starfish. The detection equipment can then transmit these underwater images to ground-based processing equipment, which inputs the images into a segmentation model to obtain the target segmentation results. Alternatively, the detection equipment can input the target images into a segmentation model to obtain the target segmentation results, and then send these results to the ground-based processing equipment.

[0046] Target segmentation by ground-based processing equipment requires the underwater detection equipment to send all acquired target images to the ground-based processing equipment. This method places lower demands on the image processing power of the underwater detection equipment, but the cost of underwater communication is relatively high. Conversely, target segmentation by the underwater detection equipment itself requires higher image processing power, but the cost of underwater communication is relatively low. The specific choice between these methods can be determined by technical personnel based on actual business needs, and will not be elaborated further here.

[0047] The following examples illustrate the business applications of target image classification results.

[0048] I. Assess the marine ecological environment.

[0049] In this case, multiple targets (usually marine organisms) can be segmented from the target image, and the marine ecological environment can be assessed based on the changes in the number and / or distribution of various targets over a period of time.

[0050] II. Assess the formation of a certain marine organism, such as coral.

[0051] In this case, a target, namely coral, can be segmented from the target image. Based on the changes in the coral's outline over a period of time, the formation of the coral can be evaluated.

[0052] It should be noted that, regardless of how many targets are segmented from the target image, it is generally necessary to determine the precise contour of each target (or segmentation target) in order to better perform the corresponding business.

[0053] The YOLOv10-seg model is currently one of the best-performing segmentation models. As shown in Figure 2, the YOLOv10-seg model generally consists of three parts: Backbone, Neck, and Head. The Backbone extracts features from the target image, obtaining feature maps with multiple feature channels, such as P3, P4, and P5 feature maps. The semantic richness of these feature maps increases sequentially, and they are all feature maps across multiple feature channels with the number of feature channels decreasing sequentially. The Neck performs bottom-up and top-down feature fusion on the feature maps with multiple feature channels, resulting in a fused feature map. The Head performs semantic parsing on the fused feature map with multiple feature channels, obtaining the target segmentation result of the target image, such as the target's contour and location.

[0054] The following is a brief introduction to the structure of the Backbone, Neck, and Head, with reference to Figure 2.

[0055] The input to the backbone is the image to be segmented. A backbone typically consists of multiple stacked feature extraction networks (C2f). Each C2f extracts features from the feature maps across multiple feature channels of the input, resulting in feature maps with a higher semantic level. C2fs are connected by convolutional layers (Conv), with the first C2f preceded by another convolutional layer. Each convolutional layer performs convolution operations on the feature maps across multiple feature channels of the input (the first convolutional layer is the input image itself). The last C2f is followed by Fast Spatial Pyramid Pooling (SPPF), which expands the receptive field of the feature maps across multiple feature channels of the input with minimal computational increase, enhancing the segmentation model's contextual awareness of multi-scale targets. The backbone outputs feature maps P3, P4, and P5 to the Neck.

[0056] The Neck can upsample (or increase dimensionality) the P5 feature map and then concatenate it with the P4 feature map. The concatenated feature map is then processed by a C2f algorithm for feature extraction. The extracted feature maps from multiple feature channels are then upsampled and concatenated with the P3 feature map. The concatenated feature map is then processed by another C2f algorithm for feature extraction. The extracted feature maps from multiple feature channels can be used as the first input to the Head (i.e., the top segment). It can also be concatenated with the feature maps from multiple feature channels output by the lower left C2f algorithm after a Conv algorithm. The concatenated feature map is then processed by another C2f algorithm for feature extraction. The extracted feature maps from multiple feature channels can be used as the second input to the Head (i.e., the middle segment). It can also be concatenated with the P5 feature map after a Conv algorithm. The concatenated feature map is then processed by another C2f algorithm for feature extraction. The extracted feature maps from multiple feature channels can be used as the third input to the Head (i.e., the bottom segment).

[0057] The head performs semantic parsing on the three segments to obtain the target segmentation result. Generally, the top segment is used to parse large targets, the middle segment is used to parse medium-sized targets, and the bottom segment is used to parse small targets.

[0058] While the YOLOv10-seg model performs well, it doesn't consider the unique characteristics of underwater target images. Directly inputting underwater target images into the YOLOv10-seg model fails to yield good segmentation results. One key reason is that underwater target images suffer from light attenuation, resulting in missing red light and retained blue-green light, along with significant noise interference. This leads to poor feature extraction after inputting the image into the segmentation model, ultimately resulting in poor target segmentation.

[0059] To this end, an enhancement network ICAM can be introduced into the backbone of the YOLOv10-seg model to enhance the color richness of target images acquired underwater. Referring to Figure 3, which is a flowchart of ICAM processing of a target image according to an embodiment of this application, the process includes the following steps.

[0060] In step 301, the target image is separated into three color channels (RGB) to obtain an initial feature map for each color channel.

[0061] Generally, target images acquired underwater are RGB images. Separating the target image into its three color channels (RGB) yields initial feature maps in the R, G, and B color channels.

[0062] In step 302, feature extraction is performed on the initial feature map of each color channel to obtain the feature map of the initial feature map of each color channel on multiple feature channels.

[0063] In practice, for the initial feature map on each color channel, semantic features can be extracted at multiple scales to obtain semantic feature maps at multiple scales (referring to different convolutional kernel sizes). For example, using 1 1, 3 3, 5 A 5x5 convolutional kernel extracts semantic features from the initial feature map to extract features of segmentation targets of different sizes. Furthermore, contour features are extracted from the initial feature map to obtain a contour feature map. Finally, the semantic feature maps, contour feature maps, and initial feature maps at multiple scales are concatenated to obtain feature maps of the initial feature map across multiple feature channels for each color channel. This approach obtains both the semantic and contour features of the initial feature map for each color channel without missing any details from the initial feature map (because it is concatenated).

[0064] In this way, extracting features from the initial feature map of each color channel separately can enhance the feature representation ability of the corresponding feature channel of each color channel for the segmentation target, thus improving the subsequent target segmentation effect.

[0065] In step 303, the initial feature map on each color channel is stitched together with the feature maps on multiple feature channels to obtain the feature map of the target image on multiple feature channels.

[0066] Assuming that each initial feature map has a feature map on N feature channels before stitching, the target image after stitching will have a feature map on 3N feature channels.

[0067] In step 304, image enhancement is performed on the target image based on the feature maps of the target image on multiple feature channels and the weight of each feature channel. The weight determination rule for each feature channel includes: the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channels corresponding to the G color channel and the B color channel.

[0068] In this way, the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channels corresponding to the G and B color channels. This can highlight the image features corresponding to the R color channel, fill in the image details lost underwater due to the lack of red light, and enhance the color balance of the target image. Therefore, it can improve the subsequent target segmentation effect.

[0069] In some embodiments, the weight determination rule for each feature channel may further include: the weight of a feature channel representing the segmentation target's features is greater than the weight of a feature channel that does not represent the segmentation target's features. That is, when a feature channel represents a feature of the segmentation target, it can be assigned a larger weight; when a feature channel does not represent a feature of the segmentation target, it can be assigned a smaller weight. This highlights the features of the segmentation target, thereby enhancing the content expression of the target image.

[0070] Generally, when enhancing a target image, the feature values ​​of the target image in the feature map of that feature channel can be adjusted according to the weight of each feature channel. For example, the weight of each feature channel can be multiplied by the feature value of the target image in the feature map of that feature channel. Then, color restoration is performed on the feature map of the target image after the feature values ​​of the feature values ​​in multiple feature channels are performed to obtain the enhanced target image (both color and content enhancement).

[0071] Referring to Figure 4, which is a schematic diagram of an ICAM structure provided in an embodiment of this application, the input is a target image acquired underwater (i.e., Image in the figure), which can be represented as 3×H×W, where 3 represents the three color channels RGB, H represents the image height, and W represents the image width. The target image can be separated in the three color channels RGB to obtain an initial feature map of 1×H×W in each color channel.

[0072] Then, the Inception Module is used to extract features from the initial 1×H×W feature map for each color channel.

[0073] For the initial 1×H×W feature map on each color channel, when extracting semantic features at different scales (corresponding to the middle two columns of the Inception Module), it can first go through 1 A convolutional kernel (Conv) is used to upscale the feature channels, expanding (or increasing) them to a 64×H×W feature map. The purpose is to decompose the different targets, their contours, and semantics from the initial 1×H×W feature map on a single color channel into different feature channels. For example, in the 64 feature channels, channels 0-15 primarily highlight contour information, channels 16-31 highlight the features of large targets, channels 32-47 highlight the features of small targets, and channels 48-63 highlight non-target features such as underwater particles and suspended matter. To better extract multi-scale target information, the following steps can be performed using 3... 3Conv、5 5. Convolutional kernels with different receptive fields, such as Conv, are used to extract features from a 64×H×W feature map. Then, 3 3Conv、5 The extraction results from Conv were processed by a leaky ReLU activation function to obtain two 64×H×W feature maps.

[0074] When extracting contour features (corresponding to the rightmost column of the Inception Module), 3 can be used 3. Pooling: The initial 1×H×W feature map on this color channel is pooled to capture the contour information of the target. The pooling result is processed by Leaky ReLU, and then... 1. After enlarging the target with Conv and then passing it through Leaky ReLU, a 64×H×W feature map is obtained.

[0075] To ensure that information in the initial 1×H×W feature map of this color channel is not overlooked, 1 can also be used 1Conv expands the initial 1×H×W feature map of this color channel into a 64×H×W feature map (corresponding to the leftmost column of the InceptionModule). Then, concatenate this 64×H×W feature map with the three previously obtained 64×H×W feature maps to obtain the 256×H×W feature map corresponding to this color channel.

[0076] Then, the 256×H×W feature maps corresponding to the three color channels are concatted to obtain the 768×H×W feature map of the target image.

[0077] It should be noted that 64, 256, and 768 above all represent the number of feature channels.

[0078] To maximize the preservation of feature information of segmented targets within different feature channels and to enhance or weaken the color expression of feature channels (for example, underwater target images appearing blue-green require reducing the weights of the G and B color channels and increasing the weight of the R color channel to restore normal color), the input feature map (i.e., a 768×H×W feature map) is fed into CAM. Parallel average pooling (Avg pooling) and max pooling (Max pooling) are then performed to extract the average pooling vector and max pooling vector for each feature channel. The average pooling vector represents the overall average statistical information of the feature map, while the max pooling vector represents the local salient feature information with the largest response value in the feature map. Next, the average pooling vector and the max pooling vector enter a multilayer perceptron (MLP). The MLP uses a two-layer fully connected network to learn the non-linear relationship between the average pooling vector and the max pooling vector, respectively, and outputs two channel weight vectors: Max Vector and Avg Vector. The two channel weight vectors are then weighted and fused to more comprehensively represent the statistical characteristics of the channels and enhance the robustness of the attention weights. The weighted and fused vector is finally passed through the sigmoid activation function in the Channel Attention module to generate attention weight coefficient vectors for each feature channel. These attention weight coefficient vectors are then multiplied by the input feature map. The process involves several steps, ultimately outputting a 768×H×W feature map. The CAM channel attention mechanism effectively enhances the overall intensity of the attenuated color gamut feature channels, highlights the feature information expression capability of the channel containing the target, weakens the response of interference information such as suspended objects and background, and effectively improves the subsequent target segmentation accuracy.

[0079] Finally, it can be done through 1 1. Conv performs feature channel dimensionality reduction on the 768×H×W feature map output by CAM to obtain an enhanced target image of 3×H×W.

[0080] In this way, while highlighting the image features corresponding to the R color channel, the expressive power of the feature channel containing the segmentation target can be enhanced, which can improve both the image color expression capability and the image content representation capability, providing a high-quality target image for subsequent target segmentation, thereby improving the target segmentation effect.

[0081] Referring to Figure 2, both Backbone and Neck use multiple stacked C2f layers for feature extraction. However, C2f involves a large number of convolutional calculations and residual structures, resulting in high computational complexity. To address the dual requirements of high-precision feature extraction and lightweight deployment for underwater target segmentation, this embodiment modifies C2f into a less computationally complex C2f-DPS (which is still a feature extraction network).

[0082] Referring to Figure 5, which is a processing flowchart of a C2f-DPS provided in an embodiment of this application, the process includes the following steps.

[0083] In step 501, convolution operations are performed on the feature maps in some of the input feature channels to obtain intermediate feature maps in multiple feature channels.

[0084] To reduce computational load, feature maps on redundant feature channels can be removed, and convolution operations can be performed only on feature maps on some feature channels, while feature maps on other feature channels are left unprocessed.

[0085] For example, if the input is a feature map with 128 feature channels, by removing redundant feature channels, only the feature maps on 64 feature channels can be convolved, while the feature maps on the other 64 feature channels are passed directly without processing. Ultimately, the result is still a feature map with 128 feature channels.

[0086] In step 502, the intermediate feature map on each feature channel is processed by a convolution kernel to obtain reference feature maps on multiple feature channels.

[0087] In this way, the intermediate feature map on each feature channel is operated with only one convolution kernel, instead of all convolution kernels, which can greatly reduce the amount of computation and thus improve the computation speed.

[0088] In step 503, the reference feature maps on multiple feature channels are reduced in dimensionality.

[0089] Generally, during dimensionality reduction, reference feature maps on unimportant or redundant feature channels are removed. This reduces the amount of data that needs to be processed, further improving computational speed.

[0090] In step 504, the feature maps on the multiple input feature channels and the reference feature maps on the multiple feature channels after dimensionality reduction are multiplied and fused to obtain the feature maps on the multiple output feature channels.

[0091] Since the preceding computational simplifications may result in the final output feature map not being able to express a rich enough range of features, the feature maps on multiple feature channels and the reference feature maps on multiple feature channels after dimensionality reduction can be multiplied and fused to expand the numerical range of the final output feature map and enable it to express a richer range of features.

[0092] Compared to C2f, C2f-DPS mainly improves the key component Bottleneck structure in C2f by transforming it into a DPSBottleneck structure. This enhances feature extraction capabilities while reducing the number of parameters and computational complexity, making it suitable for underwater target segmentation tasks that require both high accuracy and efficiency.

[0093] Referring to Figure 6, which is a schematic diagram of a C2f-DPS structure provided in an embodiment of this application, feature maps from multiple feature channels are input into the C2f-DPS. First, convolution is performed using Conv, then split using Split. After splitting, some feature maps from certain feature channels are directly passed forward, while others are fed into a stacked (e.g., n-layer) DPSBottleneck. Each DPSBottleneck extracts features from the input feature maps from multiple feature channels and outputs feature maps from multiple feature channels. Finally, the feature maps from multiple feature channels obtained in each step are concatted, and after concatting, they are output after another Conv.

[0094] The feature extraction process for each DSBottleneck is described below.

[0095] Each DPSBottleneck first performs partial convolution (Pconv) on the feature maps of multiple input feature channels, then performs depthwise convolution (DWConv), and finally performs pointwise convolution (PWConv). The resulting feature maps of multiple feature channels are then multiplied by the input feature maps of multiple feature channels. (or star operation), to obtain feature maps on multiple feature channels of the output.

[0096] The advantages of using PConv instead of regular convolution Conv are:

[0097] a) Redundant feature filtering: The feature maps of underwater target images contain highly similar feature channels such as background noise and texture repetition regions. PConv only applies convolution operations to some input channels, while keeping the remaining channels unchanged, which can effectively reduce the amount of computation.

[0098] b) Computational efficiency optimization: By reducing the number of convolution channels, PConv reduces the computational cost from... Reduce to Meanwhile, the robustness of the model is improved by retaining some original channel information. The number of feature channels in the input feature map. Where K is the number of feature channels in the output feature map, H is the kernel size, and W is the height of the input feature map. The compression ratio is the percentage of the number of selected feature channels out of the total number of feature channels.

[0099] c) Dropout concept: PConv's random channel selection mechanism is similar to the Dropout operation, which improves the model's generalization ability by suppressing redundant features.

[0100] Furthermore, DWConv and PWConv are used instead of conventional convolutional Conv. DWConv can extract spatial features independently for each feature channel, preserving the independence between feature channels. PWConv, on the other hand, extracts features through a single pass through each channel. 1. Convolution aggregates features from all feature channels, enabling interaction between feature channels. Combining DWConv and PWConv can reduce the computational cost of regular convolution from... Reduced to W significantly reduces the number of parameters and computational load.

[0101] Furthermore, star operations are used instead of point-by-point addition in the residual structure. Since star operations achieve a non-linear mapping from low-dimensional space to high-dimensional features through element-wise multiplication, they enhance feature representation capabilities. Compared to the linear addition of residual connections, the non-linear characteristics of star operations can capture more complex feature interactions, making them particularly suitable for feature enhancement in underwater target segmentation scenarios with complex backgrounds and blurred target-noise boundaries. With the same computational cost, non-linear mapping can significantly improve the feature extraction capability and final segmentation effect of the segmentation model.

[0102] In this embodiment, after replacing C2f with C2f_DPS, feature extraction is performed through multiple stacked C2f_DPS layers to obtain feature maps with various feature channel numbers, such as P3, P4, and P5 feature maps. Specifically, the feature maps input to the first-layer C2f_DPS are obtained by convolving the enhanced target image, while the feature maps input to non-first-layer C2f_DPS are obtained by convolving the feature maps output from the upper-layer C2f_DPS layers. Then, the target segmentation result can be determined based on the feature maps with various feature channel numbers.

[0103] From the perspective of the model network structure, the P3, P4, and P5 feature maps gradually transition from local feature information such as contours and textures to global information such as target category and attributes. However, each feature map cannot simultaneously capture both local and global features. Therefore, a feature fusion network (GMA) can be designed to fuse the P3, P4, and P5 feature maps separately. Subsequently, based on the fused P3, P4, and P5 feature maps, the target segmentation result is determined.

[0104] Referring to Figure 7, which is a flowchart of a GMA processing method provided in an embodiment of this application, the method includes the following steps.

[0105] In step 701, the feature maps for each number of feature channels are grouped by feature channels.

[0106] Taking the P3 feature map as an example, assuming that the P3 feature map is 64×H×W (i.e., the number of feature channels is 64), and assuming that each group has 8 feature channels, then 64×H×W can be divided into 8 groups, and each group is a feature map of 8×H×W.

[0107] In step 702, semantic features of each set of feature maps are extracted in the X and Y axis directions, and contour features of each set of feature maps are extracted.

[0108] In this way, semantics are extracted and contours are extracted group by group along the X and Y axes, which can obtain fine-grained semantic and contour information, and also retain the approximate location information of the segmentation target.

[0109] In step 703, each set of feature maps is fused with the corresponding semantic features and contour features to obtain a fused feature map for each set of feature maps.

[0110] In this way, after information fusion of each set of feature maps, we can obtain more granular contour and semantic information, and retain the approximate location information of the segmented target, as well as the original feature map information, providing location guidance for subsequent target segmentation.

[0111] In step 704, the fused feature maps corresponding to each group of feature maps are spliced ​​together to obtain a fused feature map for each number of feature channels.

[0112] Taking the P3 feature map as an example again, if the original size of the P3 feature map is 64×H×W, then the size of the P3 feature map after feature fusion is still 64×H×W.

[0113] Referring to Figure 8, Figure 8 is a schematic diagram of a GMA structure provided in an embodiment of this application. The input of the GMA is... Taking the feature map as an example, the feature channels can be grouped first to effectively reduce the amount of parameter calculation. Assuming that without grouping, after 3... When performing calculations with 3 convolutional kernels, each kernel needs to have the same number of feature channels as the output feature map. After the calculations are completed, the feature maps are merged. The computational cost is... After feature channel grouping, each input feature map is... Where g represents the number of groups, and 3 3. When calculating the convolution kernel, the computational cost of each group of parameters is ( ) ( )= ( The total computational cost for g groups is ( Compared to the non-grouping method, the computational load is reduced by g times.

[0114] Next, the feature maps on each feature channel (which can be simply referred to as each feature map group) are processed separately through three branches. The size of each feature map group is [size missing]. Specifically, the two branches on the left introduce independent Avg Pooling operations for the X and Y axes to extract high-level semantic information from this set of feature maps in the horizontal and vertical directions, respectively, resulting in complementary row and column feature vectors. These row and column feature vectors, in addition to capturing target category attributes, can also achieve preliminary localization of the segmented target based on their row and column positions. Then, the row and column feature vectors are respectively subjected to Conv 1... 1. Convolutional kernel computation is performed to fuse information across channels without changing the output feature channel dimension. Then, the features are processed through a sigmoid activation function to enhance the non-linear expressive power of the feature vectors. Next, the row feature vector, column feature vector, and this set of feature maps are multiplied (as shown in the diagram). The algorithm performs a 3x3 operation to obtain a feature map with global semantic information. This feature map can then be processed using BatchNorm and Softmax activation functions to ensure smooth training and fast convergence of the segmentation model. The rightmost side uses a 3x3 array with a certain receptive field. 3. Conv extracts spatial location information and local target features. After processing with BatchNorm and Softmax, it outputs feature maps containing local information (such as target location and contour). Finally, the feature maps with global semantic information and those with rich local target information are weighted (i.e., fused using the "+" in the diagram) to obtain a fused feature map for each set of feature maps. The output dimension remains the same. .

[0115] Furthermore, after obtaining g fused feature maps from g groups of feature maps through the above operations, g feature maps of size g can also be further processed. The fused feature maps are concatted, and the final output is still... The feature map (not shown in Figure 8) is used to ensure that the size of the feature map remains unchanged before and after feature fusion.

[0116] In this way, without changing the feature dimension, the feature value at each position in the fused feature map after cross-fusion can take into account both local and global features, which is beneficial for the subsequent segmentation model to accurately segment the target.

[0117] Referring to Figure 9, which is a schematic diagram of a segmentation model provided in an embodiment of this application, compared to Figure 2, an ICAM is added to the input layer of the Backbone. The ICAM highlights the image features corresponding to the R color channel and the features of the segmentation target. Feature fusion is performed on each feature map of the input Neck, so that the feature values ​​at each position in the resulting fused feature map can take into account both local structural information (i.e., local features) and deep semantic information (i.e., global features), improving the accuracy of subsequent target segmentation. Furthermore, all C2f values ​​are replaced with C2f_DPS to make the segmentation model a lightweight model suitable for deployment on underwater detection equipment. Other model processing procedures are similar to those in Figure 2 and will not be described in detail here.

[0118] In practical applications, after constructing the segmentation model shown in Figure 9, a training dataset can be obtained. For example, an underwater image dataset or an existing public dataset can be acquired, and these datasets can be converted to different formats to create a training dataset. Then, the segmentation model can be trained using the training dataset. After training, the segmentation model can be deployed to underwater detection equipment, which can then perform target segmentation on underwater images in real time or offline. The target segmentation results can be applied to decision support in scenarios such as underwater biometrics and marine ecological environment monitoring.

[0119] Based on the same technical concept, this application also provides a target segmentation device. The principle of the target segmentation device in solving the problem is similar to that of the target segmentation method described above. Therefore, the implementation of the target segmentation device can refer to the implementation of the target segmentation method, and the repeated parts will not be described again.

[0120] Figure 10 is a schematic diagram of a target segmentation device provided in an embodiment of this application, including:

[0121] Acquisition module 1001 is used to acquire target images collected from underwater.

[0122] The segmentation module 1002 is used to input the target image into the segmentation model to obtain the target segmentation result;

[0123] The segmentation model processes the target image according to the following steps:

[0124] Image enhancement is performed on the target image using an enhancement network;

[0125] Based on the enhanced target image, the target segmentation result is determined;

[0126] The enhancement network performs image enhancement according to the following steps:

[0127] The target image is separated into three color channels: red, green, and blue (RGB), to obtain an initial feature map for each color channel.

[0128] Feature extraction is performed on the initial feature map of each color channel to obtain feature maps on multiple feature channels corresponding to each color channel;

[0129] The feature maps of multiple feature channels corresponding to each color channel are stitched together to obtain the feature map of the target image in multiple feature channels.

[0130] Based on the feature maps of the target image in multiple feature channels and the weight of each feature channel, image enhancement is performed on the target image. The weight determination rule for each feature channel includes: the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channels corresponding to the G color channel and the B color channel.

[0131] In some embodiments, the weight determination rule for each feature channel further includes: the weight of the feature channel that represents the segmentation target feature is greater than the weight of the feature channel that does not represent the segmentation target feature.

[0132] In some embodiments, the segmentation module 1002 extracts feature maps on multiple feature channels corresponding to each color channel according to the following steps:

[0133] Semantic features are extracted at multiple scales from the initial feature map on each color channel to obtain semantic feature maps at multiple scales;

[0134] Contour feature extraction is performed on the initial feature map to obtain a contour feature map;

[0135] The semantic feature maps at various scales, the contour feature maps, and the initial feature maps are concatenated to obtain feature maps on multiple feature channels corresponding to each color channel.

[0136] In some embodiments, the segmentation module 1002 is specifically used for:

[0137] Based on the weight of each feature channel, the feature values ​​of the target image in the feature map of the feature channel are adjusted;

[0138] Color restoration is performed on the feature map after feature value adjustment on multiple feature channels of the target image to obtain the enhanced target image.

[0139] In some embodiments, the segmentation module 1002 is specifically used for:

[0140] The target image after image enhancement is subjected to feature extraction by multiple stacked feature extraction networks to obtain feature maps with multiple feature channels. The feature map input to the first layer feature extraction network is obtained by performing convolution operation on the target image after image enhancement.

[0141] The target segmentation result is determined based on the feature map of the multiple feature channels;

[0142] Each feature extraction network extracts features according to the following steps:

[0143] Convolution operations are performed on the feature maps of some feature channels of the input to obtain intermediate feature maps;

[0144] The intermediate feature map on each feature channel is processed by a convolution kernel to obtain reference feature maps on multiple feature channels;

[0145] The feature maps on multiple feature channels of the input and the reference feature maps on multiple feature channels after dimensionality reduction are multiplied and fused to obtain the feature maps on multiple feature channels of the output.

[0146] In some embodiments, the segmentation module 1002 is specifically used for:

[0147] The feature map of each feature channel number is fused by a feature fusion network to obtain a fused feature map of each feature channel number;

[0148] The target segmentation result is determined based on the fused feature map with multiple feature channels.

[0149] The feature fusion network performs feature fusion according to the following steps:

[0150] Feature maps of each feature channel number are grouped by feature channel;

[0151] Extract semantic features of each feature map along the X and Y axes, and extract contour features of each feature map.

[0152] Each set of feature maps is fused with its corresponding semantic features and contour features to obtain a fused feature map for each set of feature maps.

[0153] The fused feature maps corresponding to each set of feature maps are spliced ​​together to obtain a fused feature map for each feature channel number.

[0154] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. Coupling between modules can be achieved through interfaces, typically electrical communication interfaces, but mechanical interfaces or other types of interfaces are also possible. Therefore, modules described as separate components may or may not be physically separate; they can be located in one place or distributed across different locations on the same or different devices. The integrated modules described above can be implemented in hardware or as software functional modules.

[0155] Having introduced the target segmentation method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0156] The electronic device 130 implemented according to this embodiment of the present application will now be described with reference to FIG11. The electronic device 130 shown in FIG11 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present application.

[0157] As shown in Figure 11, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0158] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0159] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0160] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0161] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0162] In an exemplary embodiment, the electronic device of this application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, it enables the at least one processor to perform the steps of any target segmentation method provided in the embodiments of this application.

[0163] In an exemplary embodiment, a storage medium is also provided, which, when executed by a processor of an electronic device, enables the electronic device to perform any of the target segmentation methods described above. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0164] In an exemplary embodiment, a computer program product is also provided, which, when executed by an electronic device, enables the electronic device to implement any of the exemplary methods provided in this application.

[0165] It should be noted that although several modules or sub-modules of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0166] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0167] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0169] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, then this application also includes such modifications and variations.

Claims

1. A target segmentation method, characterized in that, include: The target image acquired underwater is input into the segmentation model to obtain the target segmentation result; The segmentation model processes the target image according to the following steps: image enhancement of the target image through an enhancement network; Based on the enhanced target image, the target segmentation result is determined; wherein, the enhancement network performs image enhancement according to the following steps: separating the target image in the red, green, and blue (RGB) color channels to obtain an initial feature map for each color channel; extracting features from the initial feature maps in each color channel to obtain feature maps in multiple feature channels corresponding to each color channel; concatenating the feature maps in multiple feature channels corresponding to each color channel to obtain feature maps of the target image in multiple feature channels; and enhancing the target image based on the feature maps of the target image in multiple feature channels and the weights of each feature channel, wherein the weight determination rule for each feature channel includes: the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G and B color channels; wherein, based on the enhanced target image, the target image is... Determining the target segmentation result includes: extracting features from the enhanced target image through multiple stacked feature extraction networks to obtain feature maps with multiple feature channels, wherein the feature map input to the first layer of the feature extraction network is obtained by performing a convolution operation on the enhanced target image; determining the target segmentation result based on the feature maps with multiple feature channels; wherein each feature extraction network performs feature extraction according to the following steps: performing a convolution operation on the feature maps on some of the input feature channels to obtain intermediate feature maps; performing an operation on the intermediate feature maps on each feature channel with a convolution kernel to obtain reference feature maps on multiple feature channels; multiplying and fusing the feature maps on the multiple input feature channels and the reference feature maps on the multiple feature channels after dimensionality reduction to obtain feature maps on the multiple output feature channels; wherein, during dimensionality reduction, the reference feature maps on redundant feature channels are removed.

2. The method as described in claim 1, characterized in that, The weighting rules for each feature channel also include: the weight of the feature channel that represents the segmentation target feature is greater than the weight of the feature channel that does not represent the segmentation target feature.

3. The method as described in claim 1 or 2, characterized in that, The following steps are used to extract feature maps on multiple feature channels corresponding to each color channel: semantic feature extraction at multiple scales is performed on the initial feature map on each color channel to obtain semantic feature maps at multiple scales; contour feature extraction is performed on the initial feature map to obtain contour feature maps; the semantic feature maps at multiple scales, the contour feature maps, and the initial feature maps are concatenated to obtain feature maps on multiple feature channels corresponding to each color channel.

4. The method as described in claim 1 or 2, characterized in that, Image enhancement is performed on the target image based on the feature maps of the target image across multiple feature channels and the weights of each feature channel, including: adjusting the feature values ​​of the target image in the feature maps of the feature channels according to the weights of each feature channel; and restoring the color based on the feature maps of the target image after the feature values ​​of the feature maps across multiple feature channels are adjusted, to obtain the enhanced target image.

5. The method as described in claim 1, characterized in that, The target segmentation result is determined based on the feature maps of the various feature channel numbers, including: fusing the feature maps of each feature channel number using a feature fusion network to obtain a fused feature map of each feature channel number; and determining the target segmentation result based on the fused feature maps of the various feature channel numbers. The feature fusion network performs feature fusion according to the following steps: grouping the feature maps of each feature channel number into feature channels; extracting semantic features of each group of feature maps in the X and Y axis directions, and extracting contour features of each group of feature maps; fusing each group of feature maps with its corresponding semantic features and contour features to obtain a fused feature map corresponding to each group of feature maps; and concatenating the fused feature maps corresponding to each group of feature maps to obtain a fused feature map of each feature channel number.

6. A target segmentation device, characterized in that, include: The acquisition module is used to acquire target images collected from underwater. The segmentation module is used to input the target image into the segmentation model to obtain the target segmentation result; The segmentation model processes the target image according to the following steps: image enhancement of the target image through an enhancement network; Based on the enhanced target image, the target segmentation result is determined; wherein, the enhancement network performs image enhancement according to the following steps: separating the target image in the red, green, and blue (RGB) color channels to obtain an initial feature map for each color channel; extracting features from the initial feature maps in each color channel to obtain feature maps in multiple feature channels corresponding to each color channel; concatenating the feature maps in multiple feature channels corresponding to each color channel to obtain feature maps of the target image in multiple feature channels; and enhancing the target image based on the feature maps of the target image in multiple feature channels and the weights of each feature channel, wherein the weight determination rule for each feature channel includes: the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G and B color channels; wherein, the segmentation module has The method is used to: extract features from the enhanced target image through multiple stacked feature extraction networks to obtain feature maps with multiple feature channels, wherein the feature map input to the first layer of the feature extraction network is obtained by performing a convolution operation on the enhanced target image; determine the target segmentation result based on the feature maps with multiple feature channels; wherein each feature extraction network performs feature extraction according to the following steps: performing a convolution operation on the feature maps on some of the input feature channels to obtain intermediate feature maps; performing an operation on the intermediate feature maps on each feature channel with a convolution kernel to obtain reference feature maps on multiple feature channels; multiplying and fusing the feature maps on multiple input feature channels and the reference feature maps on multiple feature channels after dimensionality reduction to obtain feature maps on multiple output feature channels; wherein, during dimensionality reduction, the reference feature maps on redundant feature channels are removed.

7. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein: the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-5.

8. A storage medium, characterized in that, When the computer program in the storage medium is executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images