Target segmentation method and device, electronic equipment and storage medium

Abstract

The invention discloses a target segmentation method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing. According to the method, for a target image collected underwater, a segmentation model carries out image enhancement through an enhancement network and then carries out target segmentation. The enhancement network separates the target image on RGB color channels, performs feature extraction on initial feature maps on each color channel after separation, and splices the extracted feature maps to obtain feature maps of the target image on multiple feature channels; and image enhancement is performed on the target image based on the feature maps of the target image on the plurality of feature channels and the weight of each feature channel, and the weight determination rule comprises that the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channel corresponding to the G color channel and the weight of the feature channel corresponding to the B color channel. Therefore, the feature information of the R color channel can be highlighted, image details lost due to red light deficiency underwater can be filled, and the precision of subsequent target segmentation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a target segmentation method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In the field of marine ecological research, it is often necessary to perform target segmentation on target images collected underwater to determine the precise contours of marine organisms such as shells, corals, fish, etc. in the target images.

[0003] Traditional target segmentation schemes rely mainly on segmentation models such as YOLOv10-seg, but they are not optimized for the particularities of underwater environments, and direct use can have some defects. For example, target images collected underwater have missing red light and residual blue-green light due to light attenuation, and have serious noise interference, which can result in poor feature extraction by the segmentation model and thus poor target segmentation results. SUMMARY

[0004] The embodiments of the present application provide a target segmentation method and device, an electronic device, and a storage medium to improve the target segmentation effect of target images collected underwater.

[0005] In a first aspect, the embodiments of the present application provide a target segmentation method, comprising: inputting a target image collected underwater into a segmentation model to obtain a target segmentation result; the segmentation model processes the target image according to the following steps: performing image enhancement on the target image by an enhancement network; determining the target segmentation result based on the image-enhanced target image; wherein the enhancement network performs image enhancement according to the following steps: separating the target image on three color channels of red, green, and blue (RGB) to obtain an initial feature map on each color channel; performing feature extraction on the initial feature map on each color channel to obtain a feature map on a plurality of feature channels corresponding to each color channel; stitching the feature maps on the plurality of feature channels corresponding to each color channel to obtain a feature map of the target image on the plurality of feature channels; performing image enhancement on the target image based on the feature map of the target image on the plurality of feature channels and the weight of each feature channel, wherein the weight determination rule of each feature channel includes that the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel.

[0006] In a second aspect, the embodiments of the present application provide a target segmentation device, comprising: an acquisition module configured to acquire a target image collected underwater; a segmentation module configured to input the target image into a segmentation model to obtain a target segmentation result; The segmentation model processes the target image according to the following steps: performing image enhancement on the target image by using an enhancement network; determining the target segmentation result based on the target image after image enhancement; The enhancement network performs image enhancement according to the following steps: separating the target image on three color channels of red, green and blue (RGB) to obtain an initial feature map on each color channel; performing feature extraction on the initial feature map on each color channel to obtain a feature map on a plurality of feature channels corresponding to each color channel; stitching the feature maps on the plurality of feature channels corresponding to each color channel to obtain a feature map of the target image on the plurality of feature channels; performing image enhancement on the target image based on the feature map of the target image on the plurality of feature channels and the weight of each feature channel, wherein the weight determination rule of each feature channel comprises that the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel.

[0007] In a third aspect, an electronic device is provided, comprising at least one processor, and a memory connected to the at least one processor in communication, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute any of the above target segmentation methods.

[0008] In a fourth aspect, an embodiment of the present application provides a storage medium, when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute any of the above target segmentation methods.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program, which, when executed by a processor, implements any of the above target segmentation methods.

[0010] In the embodiments of the present application, for the target image collected underwater, the segmentation model first performs image enhancement on the target image through the built-in enhancement network, and then performs target segmentation based on the enhanced target image. The processing flow of the enhancement network includes: separating the target image on the RGB three color channels to obtain the initial feature map on each color channel; performing feature extraction on the initial feature map on each color channel respectively to obtain the feature map on the multiple feature channels corresponding to each color channel; then splicing the multiple channel feature maps corresponding to each color channel to obtain the feature map of the target image on the multiple feature channels, and then performing image enhancement on the target image based on the feature map of the target image on the multiple feature channels and the weight of each feature channel, wherein the weight of each feature channel includes the weight of the feature channel corresponding to the R color channel, which is greater than the weight of the feature channel corresponding to the G color channel and the B color channel. In this way, performing feature extraction on the initial feature map on each color channel separately can enhance the feature representation capability of each color channel for the segmentation target; and giving a higher weight to the R channel can effectively highlight the image features related to red light, compensate for the details lost due to red light attenuation in the underwater environment, thereby enhancing the color balance and content expression capability of the target image, providing a high-quality input image for subsequent target segmentation, and thus improving the segmentation precision. In addition, embedding the enhancement network into the segmentation model can be jointly trained with the segmentation model, without the need for additional image preprocessing outside the segmentation model, realizing end-to-end optimization, and significantly improving the adaptability and processing efficiency of the segmentation model for underwater images. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings: Figure 1 An application scenario diagram of a target segmentation method provided by the embodiments of the present application; Figure 2 A structure diagram of a YOLOv10-seg model provided by the embodiments of the present application; Figure 3 A processing flow diagram of an ICAM for a target image provided by the embodiments of the present application; Figure 4 A structure diagram of an ICAM provided by the embodiments of the present application; Figure 5 A processing flow diagram of a C2f-DPS provided by the embodiments of the present application; Figure 6 A structure diagram of a C2f-DPS provided by the embodiments of the present application; Figure 7 A processing flow diagram of a GMA provided by the embodiments of the present application; Figure 8 A structure diagram of a GMA provided for an embodiment of the present application is shown in FIG. 1. Figure 9 A structure diagram of a segmentation model provided for an embodiment of the present application is shown in FIG. 2. Figure 10 A structure diagram of a target segmentation device provided for an embodiment of the present application is shown in FIG. 3. Figure 11 A hardware structure diagram of an electronic device for implementing a target segmentation method provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0012] In order to improve the target segmentation effect of the target image collected from underwater, the present application provides a target segmentation method, device, electronic device and storage medium.

[0013] Reference is made to Figure 1 , Figure 1 An application scenario diagram of a target segmentation method provided for an embodiment of the present application is shown in FIG. 5. The ground processing device is located on the ground, and the underwater detection device is located underwater. The ground processing device and the underwater detection device can communicate through the Internet.

[0014] Generally, the underwater detection device has a camera. The camera can be used to collect a target image from underwater. The target image can include various marine organisms, such as fish, coral, turtles, seaweed, crabs, starfish, etc. The detection device can send the target image collected from underwater to the ground processing device. The ground processing device inputs the target image into a segmentation model to obtain a target segmentation result. The detection device can also input the target image into the segmentation model to obtain the target segmentation result, and then send the target segmentation result to the ground processing device.

[0015] The target segmentation method performed by the ground processing device requires the detection device to send all the collected target images to the ground processing device. The image processing energy requirement of the underwater detection device is relatively low, but the cost of underwater communication is relatively high. The target segmentation method performed by the underwater detection device requires relatively high image processing energy of the underwater detection device, but the cost of underwater communication is relatively low. The specific selection of the method can be determined by the technical personnel according to the actual business requirements, which will not be described here.

[0016] The business use of the target classification result of the target image is exemplarily introduced below.

[0017] I. Evaluate the marine ecological environment.

[0018] In this case, various targets (generally marine organisms) can be segmented from the target image. According to the number change and / or distribution of various targets within a period of time, the marine ecological environment can be evaluated.

[0019] II. Evaluate the generation of a certain marine organism such as coral.

[0020] In this case, a target, i.e. coral, can be segmented from a target image, and the generation of the coral can be evaluated according to the contour change of the coral in a period of time.

[0021] It should be noted that no matter how many targets are segmented from the target image, the accurate contour of each target (or segmented target) generally needs to be determined in order to better perform the corresponding business.

[0022] The YOLOv10-seg model is a relatively good segmentation model at present, such as Figure 2 As shown in the figure, the YOLOv10-seg model generally includes Backbone, Neck and Head. Backbone is used for feature extraction of the target image to obtain feature maps with multiple feature channel numbers such as P3 feature map, P4 feature map and P5 feature map. The semantic richness of these feature maps increases in turn, and they are all feature maps on multiple feature channels with feature channel numbers decreasing in turn. Neck is used for feature fusion of the feature maps with multiple feature channel numbers from bottom to top and from top to bottom to obtain fusion feature maps with multiple feature channel numbers. Head is used for semantic analysis of the fusion feature maps with multiple feature channel numbers to obtain target segmentation results of the target image, such as the contour of the target, the position of the target, etc.

[0023] The structure of Backbone, Neck and Head will be briefly introduced below. Figure 2

[0024] The input (Input) of Backbone is the image to be segmented. Backbone generally includes a plurality of feature extraction networks C2f stacked in layers, and each C2f is used for feature extraction of the feature map on the input multiple feature channels to obtain a feature map on multiple feature channels with higher semantic levels. The C2fs can be connected through convolution layers (Conv). The first C2f is also connected in front of the convolution layer. Each convolution layer is used for convolution operation on the feature map on the input multiple feature channels (the first convolution layer is on the input image). The last C2f is connected with a fast spatial pyramid pooling (SPPF) behind it, and the SPPF is used to expand the receptive field of the feature map on the input multiple feature channels and enhance the context perception ability of the segmentation model for multi-scale targets without increasing the calculation amount. Backbone outputs P3 feature map, P4 feature map and P5 feature map to Neck.

[0025] ​The neck can upsample (or dimension up) the P5 feature map and concatenate it with the P4 feature map. The concatenated feature map is subjected to a C2f for feature extraction. The feature maps on multiple feature channels extracted are upsampled and concatenated with the P3 feature map. The concatenated feature map is subjected to a C2f for feature extraction. The feature maps on multiple feature channels extracted can be used as the first input (i.e., the uppermost segment) of the head. The feature maps on multiple feature channels extracted can also be concatenated with the feature maps on multiple feature channels output by the C2f on the left lower side after a convolution. The concatenated feature map is subjected to a C2f for feature extraction. The feature maps on multiple feature channels extracted can be used as the second input (i.e., the middle segment) of the head. The feature maps on multiple feature channels extracted can also be concatenated with the P5 feature map after a convolution. The concatenated feature map is subjected to a C2f for feature extraction. The feature maps on multiple feature channels extracted can be used as the third input (i.e., the lowermost segment) of the head.

[0026] The head performs semantic analysis on the three segments to obtain a target segmentation result. Generally, the uppermost segment is used to analyze large-size targets, the middle segment is used to analyze medium-size targets, and the lowermost segment is used to analyze small-size targets.

[0027] Although the YOLOv10-seg model has good performance, it does not consider the particularity of the target image collected underwater. Directly inputting the target image collected underwater into the YOLOv10-seg model cannot obtain good segmentation results. One important reason is that the target image collected underwater lacks red light due to light attenuation, retains blue and green light, and has serious noise interference. After being input into the segmentation model, the feature extraction effect is not good, which further leads to poor target segmentation effect.

[0028] Therefore, an enhancement network ICAM can be introduced at the head of the Backbone of the YOLOv10-seg model to enhance the color richness of the target image collected underwater. Referring to Figure 3 , Figure 3 A flowchart of a processing procedure of an ICAM for a target image provided by an embodiment of the present application includes the following steps.

[0029] In step 301, the target image is separated on the RGB three color channels to obtain an initial feature map on each color channel.

[0030] Generally, the target image collected from underwater is an RGB image. Separating the target image on the RGB three color channels will obtain an initial feature map on the R color channel, an initial feature map on the G color channel, and an initial feature map on the B color channel.

[0031] In step 302, feature extraction is performed on the initial feature map on each color channel to obtain a feature map of the initial feature map on each color channel on a plurality of feature channels.

[0032] In implementation, for the initial feature map on each color channel, semantic feature extraction can be performed on the initial feature map at multiple scales to obtain semantic feature maps at multiple scales (i.e., different convolution kernel sizes), such as performing semantic feature extraction on the initial feature map using convolution kernels of sizes 1, 3, 5, and 7, respectively, to extract features of segmentation targets of different sizes from the initial feature map. In addition, contour feature extraction can be performed on the initial feature map to obtain a contour feature map. Finally, the semantic feature maps at multiple scales, the contour feature map, and the initial feature map are spliced to obtain a feature map of the initial feature map on each color channel on a plurality of feature channels. In this way, semantic features and contour features of the initial feature map on each color channel can be obtained without missing details of the initial feature map (because the initial feature map is spliced). 1、3 3、5 5. In addition, contour feature extraction can be performed on the initial feature map to obtain a contour feature map. Finally, the semantic feature maps at multiple scales, the contour feature map, and the initial feature map are spliced to obtain a feature map of the initial feature map on each color channel on a plurality of feature channels. In this way, semantic features and contour features of the initial feature map on each color channel can be obtained without missing details of the initial feature map (because the initial feature map is spliced).

[0033] In this way, feature extraction is performed on the initial feature map on each color channel separately, which can enhance the feature expression capability of each color channel corresponding feature channel to the segmentation target, and thus improve the subsequent target segmentation effect.

[0034] In step 303, the feature maps of the initial feature map on each color channel on a plurality of feature channels are spliced to obtain a feature map of the target image on a plurality of feature channels.

[0035] Suppose that each initial feature map has feature maps on N feature channels before splicing, then the target image has feature maps on 3N feature channels after splicing.

[0036] In step 304, based on the feature map of the target image on a plurality of feature channels and the weight of each feature channel, image enhancement is performed on the target image, wherein the weight determination rule of each feature channel includes that the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel.

[0037] In this way, the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel, the image features corresponding to the R color channel can be highlighted, the image details lost due to the lack of red light underwater can be filled in, the color balance of the target image can be enhanced, and therefore, the subsequent target segmentation effect can be improved.

[0038] In some embodiments, the weight determination rule of each feature channel can further include that the weight of the feature channel representing the feature of the segmentation target is greater than the weight of the feature channel not representing the feature of the segmentation target. That is, when a feature channel represents the feature of the segmentation target, a greater weight can be given to the feature channel, and when a feature channel does not represent the feature of the segmentation target, a smaller weight can be given to the feature channel. In this way, the feature of the segmentation target can be highlighted, and the content expression of the target image can be enhanced.

[0039] Generally, when the target image is image enhanced, the feature value in the feature map of the target image on each feature channel can be adjusted according to the weight of each feature channel, for example, the weight of each feature channel is multiplied by the feature value in the feature map of the target image on the feature channel. Then, the feature map of the target image after the feature values on the multiple feature channels are adjusted is color restored, and the target image after the image enhancement (color enhancement and content enhancement) is obtained.

[0040] Referring to Figure 4 , Figure 4 A structure schematic diagram of an ICAM provided by an embodiment of the present application is shown in FIG. 1. The input is a target image collected underwater (i.e., Image in the figure), which can be represented as 3×H×W, where 3 represents three color channels of RGB, H represents the image height, and W represents the image width. The target image can be first separated on the three color channels of RGB to obtain an initial feature map of 1×H×W on each color channel.

[0041] Then, the initial feature map of 1×H×W on each color channel is subjected to feature extraction through an Inception module (Inception Module).

[0042] For the initial feature map of 1×H×W on each color channel, when different scale semantic feature extraction is performed (corresponding to the middle two columns of the Inception Module), the initial feature map can be first subjected to 1 1 Conv is used to expand the feature channel dimension, which is expanded (or called dimensioned) to 64xHxW feature map. The purpose is to disassemble different targets, target outlines, target semantics, etc. in the initial 1xHxW feature map on a single color channel and express them in different feature channels. For example, the first 0-15 feature channels of the 64 feature channels mainly highlight the outline information, the 16-31 feature channels highlight the feature information of large-size targets, the 32-47 feature channels highlight the feature information of small-size targets, and the 48-63 feature channels highlight the feature information of non-targets such as underwater particulate suspensions. In order to better extract multi-scale target information, next, 3 3 Conv, 5 5 Conv, etc. are used to extract features from the 64xHxW feature map. Then, 3 3 Conv, 5 The extraction results of 5 Conv are respectively processed by a leaky activation function (Leaky ReLU) to obtain two 64xHxW feature maps.

[0043] When extracting the outline feature (corresponding to the rightmost column of the Inception Module), 3 3 pooling is used to pool the initial 1xHxW feature map on this color channel to capture the outline information of the target. The pooling result is processed by Leaky ReLU, and then processed by 1 1 Conv to enlarge the target and then processed by Leaky ReLU to obtain a 64xHxW feature map.

[0044] In order not to miss the information in the initial 1xHxW feature map on this color channel, 1 1 Conv is used to expand the initial 1xHxW feature map on this color channel to a 64xHxW feature map (corresponding to the leftmost column of the Inception Module). Then, the 64xHxW feature map is concatenated with the three 64xHxW feature maps obtained above to obtain a 256xHxW feature map corresponding to this color channel.

[0045] Then, the three 256xHxW feature maps corresponding to the three color channels are concatenated to obtain a 768xHxW feature map of the target image.

[0046] It should be noted that the above 64, 256, and 768 represent the number of feature channels.

[0047] To maximize the preservation of feature information of segmented targets within different feature channels and to enhance or weaken the color expression of feature channels (for example, underwater target images appearing blue-green require reducing the weights of the G and B color channels and increasing the weight of the R color channel to restore normal color), the input feature map (i.e., a 768×H×W feature map) is fed into CAM. Parallel average pooling (Avg pooling) and max pooling (Max pooling) are then performed to extract the average pooling vector and max pooling vector for each feature channel. The average pooling vector represents the overall average statistical information of the feature map, while the max pooling vector represents the local salient feature information with the largest response value in the feature map. Next, the average pooling vector and the max pooling vector enter a multilayer perceptron (MLP). The MLP uses a two-layer fully connected network to learn the non-linear relationship between the average pooling vector and the max pooling vector, respectively, and outputs two channel weight vectors: Max Vector and Avg Vector. The two channel weight vectors are then weighted and fused to more comprehensively represent the statistical characteristics of the channels and enhance the robustness of the attention weights. The weighted and fused vector is finally passed through the sigmoid activation function in the Channel Attention module to generate attention weight coefficient vectors for each feature channel. These attention weight coefficient vectors are then multiplied by the input feature map. The process involves several steps, ultimately outputting a 768×H×W feature map. The CAM channel attention mechanism effectively enhances the overall intensity of the attenuated color gamut feature channels, highlights the feature information expression capability of the channel containing the target, weakens the response of interference information such as suspended objects and background, and effectively improves the subsequent target segmentation accuracy.

[0048] Finally, it can be done through 1 1. Conv performs feature channel dimensionality reduction on the 768×H×W feature map output by CAM to obtain an enhanced target image of 3×H×W.

[0049] In this way, while highlighting the image features corresponding to the R color channel, the expressive power of the feature channel containing the segmentation target can be enhanced, which can improve both the image color expression capability and the image content representation capability, providing a high-quality target image for subsequent target segmentation, thereby improving the target segmentation effect.

[0050] See Figure 2 Backbone and Neck both use multiple stacked C2f networks for feature extraction, but C2f involves a large number of convolutional calculations and residual structures, resulting in high computational complexity. To address the dual requirements of high-precision feature extraction and lightweight deployment for underwater target segmentation, this application's embodiment modifies C2f into a less computationally complex C2f-DPS (which is still a feature extraction network).

[0051] Referring to Figure 5 , Figure 5 A C2f-DPS processing flowchart provided by an embodiment of the present application includes the following steps.

[0052] In step 501, convolution operation is performed on the feature maps on the input partial feature channels to obtain intermediate feature maps on multiple feature channels.

[0053] In order to reduce the amount of calculation, the feature maps on the redundant feature channels can be removed, and only the feature maps on part of the feature channels are subjected to convolution operation, and the feature maps on the other part of the feature channels are not processed.

[0054] For example, the input is feature maps on 128 feature channels, and through redundant feature channel removal, only the feature maps on 64 feature channels are subjected to convolution operation, and the feature maps on the other 64 feature channels are not processed and directly passed. Finally, feature maps on 128 feature channels are still obtained.

[0055] In step 502, the intermediate feature maps on each feature channel are operated with a convolution kernel to obtain reference feature maps on multiple feature channels.

[0056] In this way, the intermediate feature maps on each feature channel are only operated with one convolution kernel, and do not have to be operated with all the convolution kernels, which can greatly reduce the amount of operation and thus improve the operation speed.

[0057] In step 503, dimensionality reduction is performed on the reference feature maps on multiple feature channels.

[0058] Generally, the reference feature maps on unimportant or redundant feature channels are removed during dimensionality reduction. In this way, the amount of data to be operated can be reduced, and the operation speed is further improved.

[0059] In step 504, the input feature maps on multiple feature channels and the reference feature maps on multiple feature channels after dimensionality reduction are multiplied and fused to obtain output feature maps on multiple feature channels.

[0060] Because of the great simplification of operation in the foregoing, the features that can be expressed by the final output feature maps may not be rich enough. Therefore, the feature maps on multiple feature channels and the reference feature maps on multiple feature channels after dimensionality reduction are multiplied and fused to expand the numerical range of the final output feature maps, so that they can express more rich features.

[0061] Compared with C2f, C2f-DPS mainly improves the bottleneck structure in C2f, improves the bottleneck structure into DPS bottleneck structure, enhances the feature extraction capability, and reduces the parameter quantity and the calculation complexity, so as to be suitable for the underwater target segmentation task which requires high precision and efficiency.

[0062] Referring to Figure 6 , Figure 6 A structure diagram of C2f-DPS provided by the embodiment of the application is shown in the figure. After the feature maps on multiple feature channels are input into C2f-DPS, the feature maps are first subjected to convolution operation through Conv, and then subjected to splitting through Split. After splitting, the feature maps on a part of the feature channels are directly transmitted backward, and the feature maps on a part of the feature channels enter the DPS bottleneck in layers (for example, n layers). Each DPS bottleneck extracts features from the feature maps on the input multiple feature channels, and outputs the feature maps on multiple feature channels. Finally, the feature maps on multiple feature channels obtained at each step are subjected to Concat, and then output after a Conv.

[0063] The feature extraction process of each DPS bottleneck is introduced below.

[0064] Each DPS bottleneck first performs partial convolution operation Pconv on the feature maps on the input multiple feature channels, then performs depthwise convolution DWConv operation, and finally performs pointwise convolution PWConv, so as to perform matrix multiplication (or star operation) on the obtained multiple feature channels and the input multiple feature channels, and obtain the output multiple feature channels.

[0065] Among them, PConv is used to replace the conventional convolution Conv, and the advantages are as follows: a) Redundant feature filtering: there are highly similar feature channels such as background noise and repeated texture regions in the feature maps of the target image collected underwater. PConv only applies convolution operation to part of the input channels, and the rest of the channels remain unchanged, which can effectively reduce the calculation amount; b) Calculation efficiency optimization: by reducing the number of convolution channels, PConv reduces the calculation amount from to , and at the same time, by retaining part of the original channel information, the model robustness is improved. Among them, is the number of feature channels of the input feature map, is the number of feature channels of the output feature map, K is the size of the convolution kernel, H is the height of the input feature map, and W is the width of the input feature map, ​to compress the ratio, i.e., the proportion of the selected feature channel number in the total feature channel number; c) Dropout idea: the random channel selection mechanism of PConv is similar to the Dropout operation, which improves the model generalization ability by suppressing redundant features.

[0066] And, DWConv and PWConv are used to replace the conventional convolution Conv. Among them, DWConv can independently extract spatial features for each feature channel, preserving the independence between feature channels. And PWConv can realize the interaction between feature channels by 1 1 convolution aggregates the features of all feature channels to realize the interaction between feature channels. The combination of DWConv and PWConv can reduce the computational complexity of conventional convolution from , to W, significantly reducing the parameter amount and computational complexity.

[0067] In addition, the star operation is used to replace the point-by-point addition (Add) in the residual structure. Since the star operation realizes nonlinear mapping from low-dimensional space to high-dimensional features through element-level multiplication, it enhances the feature representation ability. Compared with the linear addition of residual connection, the nonlinear characteristics of the star operation can capture more complex feature interaction relationships, especially suitable for feature enhancement in underwater target segmentation scenarios with complex background and blurred target and noise boundaries. Under the same computational complexity, the feature extraction ability and final segmentation effect of the segmentation model can be significantly improved through nonlinear mapping.

[0068] In the embodiments of the present application, after replacing C2f with C2f_DPS, feature extraction is performed through the stacked multiple C2f_DPS to obtain feature maps with multiple feature channel numbers, such as P3 feature map, P4 feature map and P5 feature map. Among them, the feature map on the multiple feature channels input to the first layer C2f_DPS is obtained by performing convolution operation on the target image after image enhancement, and the feature map on the multiple feature channels input to the non-first layer C2f_DPS is obtained by performing convolution operation on the feature map on the multiple feature channels output by the upper layer C2f_DPS. Then, the target segmentation result can be determined according to the feature maps with multiple feature channel numbers.

[0069] From the model network structure, the P3 feature map, the P4 feature map and the P5 feature map gradually transition from local feature information such as contour and texture to global information such as target class and attribute. However, each feature map cannot balance local features and global features, therefore, a feature fusion network GMA can be designed to perform feature fusion on the P3 feature map, the P4 feature map and the P5 feature map respectively. Subsequently, the target segmentation result is determined based on the P3 feature map, the P4 feature map and the P5 feature map after feature fusion.

[0070] Referring to Figure 7 , Figure 7A processing flowchart of the GMA provided in the embodiments of the present application includes the following steps.

[0071] In step 701, the feature maps of each feature channel number are grouped by feature channels.

[0072] Taking the P3 feature map as an example, assuming that the P3 feature map is 64xHxW (i.e., the feature channel number is 64), and assuming that there are 8 feature channels in each group, 64xHxW can be divided into 8 groups, and each group is a feature map of 8xHxW.

[0073] In step 702, the semantic features of each group of feature maps in the X-axis and Y-axis directions are extracted, and the contour features of each group of feature maps are extracted.

[0074] In this way, the semantic features and contour features are extracted in the X-axis and Y-axis directions in each group, that is, the fine-grained semantic information and contour information can be obtained, and the approximate position information of the segmentation target can also be retained.

[0075] In step 703, the feature maps in each group and the semantic features and contour features corresponding to the feature maps in each group are fused to obtain the fused feature maps corresponding to each group of feature maps.

[0076] In this way, after information fusion of each group of feature maps, more granular contour information and semantic information can be obtained, and the approximate position information of the segmentation target and the original feature map information can be retained, providing position guidance for subsequent target segmentation.

[0077] In step 704, the fused feature maps corresponding to each group of feature maps are spliced to obtain the fused feature maps of each feature channel number.

[0078] Still taking the P3 feature map as an example, if the original size of the P3 feature map is 64xHxW, the size of the P3 feature map after feature fusion is still 64xHxW.

[0079] Referring to Figure 8 , Figure 8 A structure schematic diagram of the GMA provided in the embodiments of the present application. Taking the input of the GMA as the feature map as an example, the feature channels can be grouped first to effectively reduce the parameter calculation amount. Assuming that when not grouped, 3 convolution kernels are used for calculation, each convolution kernel needs to be the same as the feature channel number of the output feature map, and after calculation, the feature maps are merged, and the calculation amount is After grouping of the feature channels, each group of the input feature maps is , where g represents the number of groups, and 3 3 convolution kernels are used for calculation, and the parameter calculation amount of each group is )​​ ), the total calculation amount of g groups is ), which is reduced by g times compared with the calculation amount without grouping.

[0080] Next, the feature map on each group of feature channels (which can be simply referred to as each group of feature maps) is processed through three branches respectively, and the size of each group of feature maps is . Specifically, the left two branches extract high-level semantic information of the group of feature maps in the horizontal direction and the vertical direction respectively by introducing Avg Pool operations independent of the X axis and the Y axis, to obtain row feature vectors and column feature vectors with complementary high-level semantic information. Among them, in addition to being able to capture the target class attributes, the row feature vectors and the column feature vectors can also achieve preliminary positioning of the segmentation target based on the row and column positions. Then, the row feature vectors and the column feature vectors are respectively subjected to Conv 1 1 convolution kernel calculation to perform information fusion across channels without changing the dimension of the output feature channel, and then pass through a sigmoid activation function respectively to improve the nonlinear expression ability of the feature vectors. Next, the row feature vectors, the column feature vectors, and the group of feature maps are multiplied (the multiplication in the figure ) to obtain feature maps with global semantic information. The feature maps with global semantic information can also be processed by BatchNorm and Softmax activation functions to ensure smooth training and fast convergence of the segmentation model. The rightmost side uses 3 3 Conv with a certain receptive field to extract spatial position information and target local features, and also outputs feature maps containing local information (such as target position, contour, etc.) after BatchNorm and Softmax processing. Finally, the feature maps with global semantic information and the feature maps with rich target local information are fused by weighting (i.e. + in the figure), to obtain a fusion feature map corresponding to each group of feature maps, and the output dimension is still .

[0081] Moreover, after g group of feature maps are obtained through the above operations to obtain g fusion feature maps with a size of , the g fusion feature maps can be concatenated, and finally a feature map with a size of is outputted (not shown in the figure) to ensure that the size of the feature map remains unchanged before and after feature fusion. Figure 8

[0082] In this way, without changing the feature dimension, the feature values at each position in the fusion feature map after cross-fusion can take into account both local features and global features, which is conducive to the subsequent segmentation model accurately performing target segmentation.

[0083] Referring to Figure 9 ,​​​Figure 9 A structure diagram of a segmentation model provided by an embodiment of the present application, compared with Figure 2 It adds ICAM to the input layer of Backbone, highlights the image features corresponding to the R color channel through ICAM, and highlights the features of the segmentation target features. And each feature map input into the neck is fused, so that the feature value at each position of the resulting fusion feature map can take into account both local structure information (i.e., local features) and deep semantic information (i.e., global features), improving the accuracy of subsequent target segmentation and. In addition, all C2f is replaced by C2f_DPS, so that the segmentation model becomes a lightweight model suitable for deployment on underwater exploration equipment. Other model processing processes are Figure 2 similar, and will not be repeated here.

[0084] In actual application, after the segmentation model shown in Figure 9 is constructed, a training data set can be obtained. For example, an underwater image data set or an existing public data set is obtained, and the data sets are format-converted to make a training data set. Then, the training data set can be used to train the segmentation model. The trained segmentation model can be deployed to the underwater exploration equipment, which performs real-time or offline target segmentation on the target image collected from the underwater environment. The target segmentation result can be applied to auxiliary decision-making in underwater biological recognition, marine ecological environment detection and other scenarios.

[0085] Based on the same technical concept, an embodiment of the present application also provides a target segmentation device. The principle of the target segmentation device to solve the problem is similar to the above-mentioned target segmentation method, so the implementation of the target segmentation device can be referred to the implementation of the target segmentation method, and the repeated parts will not be repeated.

[0086] Figure 10 A structure diagram of a target segmentation device provided by an embodiment of the present application, comprising: The acquisition module 1001 is configured to acquire a target image collected from the underwater environment. The segmentation module 1002 is configured to input the target image into a segmentation model to obtain a target segmentation result. The segmentation model processes the target image according to the following steps: The target image is image-enhanced by an enhancement network; Based on the image-enhanced target image, the target segmentation result is determined; The enhancement network performs image enhancement according to the following steps: The target image is separated on three color channels of red, green and blue (RGB) to obtain an initial feature map on each color channel; perform feature extraction on the initial feature map on each color channel to obtain a feature map on a plurality of feature channels corresponding to each color channel; splicing the feature maps on the plurality of feature channels corresponding to each color channel to obtain a feature map of the target image on the plurality of feature channels; performing image enhancement on the target image based on the feature map of the target image on the plurality of feature channels and the weight of each feature channel, wherein the weight determination rule of each feature channel comprises that the weight of the feature channel corresponding to the R color channel is greater than the weight of the feature channel corresponding to the G color channel and the B color channel.

[0087] In some embodiments, the weight determination rule of each feature channel further comprises that the weight of a feature channel representing a segmented target feature is greater than the weight of a feature channel not representing a segmented target feature.

[0088] In some embodiments, the segmentation module 1002 extracts the feature map on the plurality of feature channels corresponding to each color channel according to the following steps: performing semantic feature extraction of multiple scales on the initial feature map on each color channel to obtain semantic feature maps of multiple scales; performing contour feature extraction on the initial feature map to obtain a contour feature map; splicing the semantic feature maps of multiple scales, the contour feature map, and the initial feature map to obtain a feature map on a plurality of feature channels corresponding to each color channel.

[0089] In some embodiments, the segmentation module 1002 is specifically used for: adjusting the feature values in the feature map of the target image on the feature channel according to the weight of each feature channel; performing color restoration based on the feature map of the target image on the plurality of feature channels after the feature value adjustment to obtain the target image after image enhancement.

[0090] In some embodiments, the segmentation module 1002 is specifically used for: performing feature extraction on the target image after image enhancement through a plurality of stacked feature extraction networks to obtain feature maps of a plurality of feature channel numbers, wherein the feature map input into the first layer feature extraction network is obtained by performing convolution operation on the target image after image enhancement; determining the target segmentation result according to the plurality of feature channel number feature maps; wherein each feature extraction network performs feature extraction according to the following steps: performing convolution operation on the feature map on the input part of the feature channel to obtain an intermediate feature map; The intermediate feature map on each feature channel is operated with a convolution kernel to obtain a reference feature map on multiple feature channels; The feature map on the input multiple feature channels and the reference feature map on the multiple feature channels after dimension reduction are multiplied and fused to obtain a feature map on the output multiple feature channels.

[0091] In some embodiments, the segmentation module 1002 is specifically configured to: The feature fusion network is used to perform feature fusion on the feature map of each feature channel number to obtain a fusion feature map of each feature channel number; Based on the fusion feature maps of multiple feature channel numbers, the target segmentation result is determined; The feature fusion network performs feature fusion according to the following steps: The feature map of each feature channel number is grouped; The semantic features of each group of feature maps in the X-axis and Y-axis directions are extracted, and the contour features of each group of feature maps are extracted; Each group of feature maps and the corresponding semantic features and contour features of each group of feature maps are fused to obtain a fusion feature map corresponding to each group of feature maps; The fusion feature maps corresponding to each group of feature maps are spliced to obtain a fusion feature map of each feature channel number.

[0092] The division of the modules in the embodiments of the present application is illustrative, and is merely a logical function division. In actual implementation, another division manner can be used. In addition, the function modules in each embodiment of the present application can be integrated in one processor, or can be physically separated, or two or more modules can be integrated in one module. The coupling between the modules can be realized through some interfaces. These interfaces are usually electrical communication interfaces, but can also be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components can be or can not be physically separated, and can be located in one place or distributed to different locations of the same or different devices. The integrated modules can be realized in the form of hardware or in the form of software function modules.

[0093] After introducing the target segmentation method and device of the example embodiment of the present application, next, an electronic device according to another example embodiment of the present application is introduced.

[0094] The electronic device 130 implemented according to this embodiment of the present application is described below with reference to Figure 11 Figure 11 The displayed electronic device 130 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0095] As​Figure 11 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 can include, but are not limited to, the at least one processor 131, the at least one memory 132, and a bus 133 that connects the different system components including the memory 132 and the processor 131.

[0096] The bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor or local bus using any of a variety of bus architectures, etc.

[0097] The memory 132 can include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and can further include a read-only memory (ROM) 1323.

[0098] The memory 132 can also include a program / utility 1325 having a set of program modules 1324 including an operating system, one or more application programs, other program modules, and program data, each of which can give the electronic device 130 its functionality, or some combination thereof.

[0099] The electronic device 130 can also communicate with one or more external devices 134 such as a keyboard or a pointing device, which can be a device that enables a user to interact with the electronic device 130, and / or with any devices (such as a router, a modem, and so forth) that enable the electronic device 130 to communicate with one or more other electronic devices. Such communication can occur via an input / output (I / O) interface 135. Still yet, the electronic device 130 can communicate with one or more networks, such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet, through a network adapter 136. As depicted, the network adapter 136 communicates with the other components of the electronic device 130 via the bus 133. It should be appreciated that the network adapter 136 and / or the other hardware and / or software components depicted in FIG. 1 can be utilized in conjunction with the electronic device 130, although they are not explicitly shown. For example, a microphone can be included in the electronic device 130 to facilitate voice commands for the electronic device 130.

[0100] In an example embodiment, the electronic device of the present application can at least include at least one processor, and a memory connected with the at least one processor in communication, wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to make the at least one processor execute the steps of any target segmentation method provided by the embodiments of the present application.

[0101] In an example embodiment, a storage medium is also provided, when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform any of the above object segmentation methods. Optionally, the storage medium can be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0102] In an example embodiment, a computer program product is also provided, when the computer program product is executed by an electronic device, the electronic device can implement any of the example methods provided by the present application.

[0103] It should be noted that although several modules or sub-modules of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules embodied.

[0104] In addition, although the operations of the methods of the present application are described in a particular order in the drawings, this is not required or implied in any way as to the order of the operations or that all of the operations be performed to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or further divided into multiple steps.

[0105] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can be embodied in the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROMs, optical memory, etc.) having computer usable program code embodied thereon.

[0106] Although preferred embodiments of the present application have been described, those skilled in the art will appreciate that additional modifications and variations to the preferred embodiments can be made without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be interpreted as including all such modifications and variations as fall within the scope of the present application.

[0107] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Thus, it is intended that the present application embrace all such modifications and changes and, accordingly, the application is not to be construed as limited to the preferred embodiments.

Claims

1. A method of object segmentation, characterized by, The method comprises the steps of: inputting an underwater collected target image into a segmentation model to obtain a target segmentation result; the segmentation model processes the target image according to the following steps: performing image enhancement on the target image through an enhancement network; determining the target segmentation result based on the image-enhanced target image; wherein the enhancement network performs image enhancement according to the following steps: separating the target image on three color channels of red, green and blue (RGB) to obtain an initial feature map on each color channel; performing feature extraction on the initial feature map on each color channel to obtain a feature map on a plurality of feature channels corresponding to each color channel; splicing the feature maps on the plurality of feature channels corresponding to each color channel to obtain a feature map of the target image on the plurality of feature channels; performing image enhancement on the target image based on the feature map of the target image on the plurality of feature channels and the weight of each feature channel, wherein the weight determination rule of each feature channel comprises: the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel.

2. The method of claim 1, wherein, The weight determination rule of each feature channel further comprises: the weight of a feature channel representing a segmentation target feature is greater than the weight of a feature channel not representing a segmentation target feature.

3. The method of claim 1 or 2, wherein, The feature map on the plurality of feature channels corresponding to each color channel is extracted according to the following steps: performing semantic feature extraction of multiple scales on the initial feature map on each color channel to obtain semantic feature maps of multiple scales; performing contour feature extraction on the initial feature map to obtain a contour feature map; splicing the semantic feature maps of multiple scales, the contour feature map and the initial feature map to obtain a feature map on the plurality of feature channels corresponding to each color channel.

4. The method of claim 1 or 2, wherein, Performing image enhancement on the target image based on the feature map of the target image on the plurality of feature channels and the weight of each feature channel, comprising: adjusting the feature values in the feature map of the target image on the feature channel according to the weight of each feature channel; performing color restoration based on the feature map of the target image on the plurality of feature channels after the feature value adjustment to obtain the image-enhanced target image.

5. The method of claim 1 or 2, wherein, Determining the target segmentation result based on the image-enhanced target image, comprising: performing feature extraction on the image-enhanced target image through a plurality of stacked feature extraction networks to obtain feature maps of multiple feature channel numbers, wherein the feature map input into the first layer feature extraction network is obtained by performing convolution operation on the image-enhanced target image; determining the target segmentation result according to the feature maps of the multiple feature channel numbers; wherein each feature extraction network performs feature extraction according to the following steps: performing convolution operation on the feature map on the input part of the feature channel to obtain an intermediate feature map; performing operation on the intermediate feature map on each feature channel with a convolution kernel to obtain a reference feature map on the plurality of feature channels; performing multiplication fusion on the feature map on the input plurality of feature channels and the reference feature map on the plurality of feature channels after dimension reduction to obtain the feature map on the output plurality of feature channels.

6. The method of claim 5, wherein, Determine the target segmentation result according to the feature maps of the plurality of feature channel numbers, including: Perform feature fusion on the feature maps of each feature channel number through a feature fusion network to obtain a fusion feature map of each feature channel number; Determine the target segmentation result based on the fusion feature maps of the plurality of feature channel numbers; The feature fusion network performs feature fusion according to the following steps: Group the feature maps of each feature channel number; Extract semantic features of each group of feature maps in the X-axis and Y-axis directions, and extract contour features of each group of feature maps; Fuse each group of feature maps and the semantic features and contour features corresponding to each group of feature maps to obtain a fusion feature map corresponding to each group of feature maps; Splice the fusion feature maps corresponding to each group of feature maps to obtain a fusion feature map of each feature channel number.

7. An object segmentation apparatus characterized by comprising: Including: An acquisition module configured to acquire a target image collected underwater; A segmentation module configured to input the target image into a segmentation model to obtain a target segmentation result; The segmentation model processes the target image according to the following steps: Perform image enhancement on the target image through an enhancement network; Determine the target segmentation result based on the target image after image enhancement; The enhancement network performs image enhancement according to the following steps: Separate the target image in the red, green, and blue (RGB) three color channels to obtain an initial feature map on each color channel; Perform feature extraction on the initial feature map on each color channel to obtain a feature map on a plurality of feature channels corresponding to each color channel; Splice the feature maps on the plurality of feature channels corresponding to each color channel to obtain a feature map of the target image in the plurality of feature channels; Perform image enhancement on the target image based on the feature map of the target image in the plurality of feature channels and the weight of each feature channel, wherein the weight determination rule of each feature channel includes that the weight of the feature channel corresponding to the R color channel is greater than the weights of the feature channels corresponding to the G color channel and the B color channel.

8. An electronic device, comprising: Including: At least one processor, and a memory connected in communication with the at least one processor, wherein: The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-6.

9. A storage medium, characterized by When the computer program in the storage medium is executed by the processor of the electronic device, the electronic device can execute the method of any one of claims 1-6.

10. A computer program product, characterised in that, Including a computer program, which, when executed by a processor, implements the method of any one of claims 1-6.

Images