Image segmentation method, electronic device and computer readable storage medium
By extracting and combining multi-scale features from images, the problems of universality and computational cost of image segmentation methods on targets of different sizes are solved, and efficient image segmentation results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2026-02-04
- Publication Date
- 2026-06-05
AI Technical Summary
Existing image segmentation methods require customized feature extraction methods to adapt to targets of different sizes, resulting in excessive computational consumption and a lack of versatility.
By acquiring the original image at the largest scale and its multiple images at decreasing scales, pre-feature extraction is performed on the image at the smallest scale. Feature combination and fusion are then performed step by step. The computational power requirement is compressed by utilizing feature channels to achieve the universality of output features at each scale.
It enhances the versatility of image segmentation, reduces computational consumption, and improves the efficiency and adaptability of image segmentation.
Smart Images
Figure CN122156607A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image segmentation method, an electronic device, and a computer-readable storage medium. Background Technology
[0002] Image segmentation is a crucial aspect of image processing. It assigns a classification label to each pixel in an image, thereby separating different targets. However, adapting to targets of varying sizes often requires customized feature extraction methods at corresponding scales, consuming significant computational resources. Therefore, enhancing the versatility of image segmentation while reducing computational cost has become a pressing issue. Summary of the Invention
[0003] The main technical problem addressed by this application is to provide an image segmentation method, electronic device, and computer-readable storage medium that can enhance the versatility of image segmentation and reduce computational cost.
[0004] To address the aforementioned technical problems, this application provides an image segmentation method, comprising: acquiring multiple images of decreasing scales corresponding to the original image at the largest scale; performing pre-feature extraction on the image at the smallest scale to obtain output features after scale reduction; wherein, traversing from the smallest scale to the largest scale, each scale image is used as an input image; acquiring image features corresponding to the input image at the current scale; grouping the image features and the output features of the previous scale according to feature channels to obtain multiple feature groups and determining representative features corresponding to the feature groups; determining output features at the current scale based on each channel of the image features and all the representative features; wherein each feature group includes features of some channels of the image features or output features; and, in response to obtaining the output features corresponding to the scale of the original image, performing image segmentation using the final output features.
[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.
[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.
[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application obtains the original image at its maximum scale and multiple images at decreasing scales corresponding to the original image. Pre-feature extraction is performed on the image at the minimum scale to obtain the scale-reduced output features. Specifically, images from the minimum to the maximum scale are used sequentially as input images, and the process iterates from the minimum to the maximum scale. The decreasing scale can be set according to the target size as needed. Image features corresponding to the input image at the current scale are obtained. These image features and the output features obtained at the previous scale are grouped according to feature channels to obtain multiple feature groups. Each feature group includes features from some channels of the image features or output features. This allows for feature compression within the multiple channels corresponding to the feature group to reduce computational requirements. Representative features corresponding to the feature group are determined. Based on each channel of the image features and all representative features, the output features at the current scale are determined so that the output features can fuse features from both scales. Furthermore, feature extraction and feature fusion can be performed on each pair of scales in the same way, making the method of obtaining output features at each scale universal. When the output features corresponding to the scale of the original image are obtained, image segmentation is performed using the final output features to obtain the image segmentation result, thereby enhancing the universality of image segmentation and reducing computational cost. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one embodiment of the image segmentation method of this application; Figure 2 This is a schematic diagram illustrating an application scenario of one embodiment of the image segmentation network of this application; Figure 3 This is a flowchart illustrating another embodiment of the image segmentation method of this application; Figure 4 This is a schematic diagram illustrating an application scenario of one embodiment of the pre-feature extraction module of this application; Figure 5 This is a schematic diagram illustrating an application scenario of one implementation method of the basic multi-scale module of this application; Figure 6 This is a schematic diagram illustrating an application scenario of one implementation of the dual-scale fusion network layer of this application; Figure 7 This is a schematic diagram illustrating an application scenario of one implementation method for obtaining interleaved combination features in this application; Figure 8This is a schematic diagram illustrating an application scenario of one implementation method for obtaining dual-scale fusion features in this application; Figure 9 This is a schematic diagram illustrating an application scenario of another implementation of the pre-feature extraction module of this application; Figure 10 This is a schematic diagram illustrating an application scenario of another implementation of the basic multi-scale module of this application; Figure 11 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 12 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation
[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0010] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.
[0011] The image segmentation method provided in this application is used for image segmentation, and its corresponding execution subject is a processing unit capable of image processing.
[0012] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the image segmentation method of this application, which includes: S101: Obtain multiple images of decreasing scales corresponding to the original image at the largest scale, perform pre-feature extraction on the image at the smallest scale, and obtain the output features after scale reduction; wherein, traverse from the smallest scale to the largest scale, and use the image at each scale as the input image.
[0013] Specifically, the original image at the largest scale and multiple images at decreasing scales corresponding to the original image are obtained. Pre-feature extraction is performed on the image at the smallest scale to obtain the output features after scale reduction.
[0014] It should be noted that the images from the smallest to the largest scale are used as input images in sequence, and then the process is repeated from the smallest scale to the largest scale. The decreasing scale can be set according to the size of the target as needed.
[0015] In one embodiment, the original image at the largest scale is obtained, and the original image is downsampled multiple times using pooling operations to obtain multiple images with decreasing scales. The image at the smallest scale is obtained, and pre-feature extraction is performed on the image at the smallest scale. Downsampling is performed during the pre-feature extraction process to obtain the output features after scale reduction.
[0016] In one embodiment, the original image at the largest scale is obtained, and the original image is downsampled multiple times using bilinear interpolation to obtain multiple images with decreasing scales. The image at the smallest scale is obtained, and pre-feature extraction is performed on the image at the smallest scale. The image at the smallest scale is then reduced by a factor and its features are extracted to obtain the output features after scale reduction.
[0017] Optionally, the pooling operation includes average pooling downsampling and max pooling downsampling, and the bilinear operation includes bilinear downsampling.
[0018] It should be noted that the pre-feature extraction process is implemented by the pre-feature extraction module, which is configured with corresponding network layers and has been pre-trained.
[0019] In some implementation scenarios, the pre-feature extraction process includes performing at least one single-step convolution on the smallest scale image to obtain features at the corresponding scale, then performing pooling operations to adjust the feature scale, and performing at least one single-step convolution on the scale-adjusted features to obtain the scale-reduced output features.
[0020] In some implementation scenarios, the pre-feature extraction process includes performing at least one single-step convolution on the smallest scale image and then performing a pre-set multi-step convolution to reduce the feature scale, thereby obtaining the scaled output features.
[0021] In some implementation scenarios, the pre-feature extraction process includes downsampling the image at the smallest scale and performing at least one single-step convolution on the downsampled image to obtain the scaled output features.
[0022] S102: Obtain the image features corresponding to the input image at the current scale, group the image features and the output features of the previous scale according to the feature channels to obtain multiple feature groups and determine the representative features corresponding to the feature groups, and determine the output features at the current scale based on each channel of the image features and all representative features; wherein, each feature group includes features of some channels in the image features or output features.
[0023] Specifically, the image features corresponding to the input image at the current scale are obtained, and the image features and the output features obtained at the previous scale are grouped according to the feature channels to obtain multiple feature groups. Each feature group includes features from some channels of the image features or output features. In this way, the features are compressed in multiple channels corresponding to the feature group to reduce the computational power requirement, and the representative features corresponding to the feature group are determined.
[0024] Furthermore, based on each channel and all representative features of the image features, the output features at the current scale are determined so that the output features can be fused with features from two scales. In addition, feature extraction and feature fusion can be performed in the same way for each two scales, so that the way of obtaining output features at each scale is universal.
[0025] In one embodiment, image features corresponding to the input image at the current scale are obtained, wherein the image features have the same scale as the output features at the previous scale. The image features and the output features at the previous scale are grouped according to feature channels to obtain multiple feature groups, such that each feature group includes a preset number of channels. Representative features corresponding to each feature group are extracted. Each channel of the image features is concatenated with all representative features to perform feature extraction. The extracted features are then fused and upsampled to obtain the output features at the current scale.
[0026] In one embodiment, image features corresponding to the input image at the current scale are obtained, wherein the image features are consistent with the current scale. The output features of the previous scale are adjusted to the same scale as the image features. The image features and the output features of the previous scale are grouped according to feature channels to obtain multiple feature groups, such that each feature group includes its corresponding channel combination. Representative features corresponding to each feature group are extracted. All representative features are concatenated to obtain representative combined features. Each channel of the image features is fused with the representative combined features, and the fused features are extracted to obtain the output features at the current scale.
[0027] It should be noted that the input image is usually an RGB image, and its dimensions are typically... Here, 1 corresponds to the batch size, 3 corresponds to the feature channels (RGB three channels), H corresponds to the image height, and W corresponds to the image width. The scale corresponds to the image height and image width. The dimensions corresponding to the image features and output features are... Where N is the number of feature channels corresponding to the image features and output features, N is Typically, it is multi-channel, and image features and output features can correspond to the same number of channels or different numbers of channels.
[0028] In some implementation scenarios, a preset number of channels are extracted from the image features sequentially according to the feature channels and combined until all channels are traversed to obtain multiple feature groups corresponding to the image features. Similarly, a preset number of channels are extracted from the output features of the previous scale sequentially and combined until all channels are traversed to obtain multiple feature groups corresponding to the output features.
[0029] In some implementation scenarios, some channels are randomly extracted from the image features according to the feature channels and combined until all channels are traversed to obtain multiple feature groups corresponding to the image features. Some channels are randomly extracted from the output features of the previous scale and combined until all channels are traversed to obtain multiple feature groups corresponding to the output features.
[0030] S103: In response to obtaining the scale-corresponding output features of the original image, perform image segmentation using the final output features.
[0031] Specifically, when the output features corresponding to the scale of the original image are obtained, the final output features are used to perform image segmentation to obtain the image segmentation result, thereby enhancing the versatility of image segmentation and reducing computational power.
[0032] In one embodiment, once the output features corresponding to the scale of the original image are obtained, the final output features are input to the segmentation module, so that the segmentation module performs image segmentation based on the final output features to obtain the image segmentation result. The segmentation module is pre-trained.
[0033] In one embodiment, once the output features corresponding to the scale of the original image are obtained, the final output features are input into a large language model, so that the large language model performs image segmentation based on the final output features to obtain the image segmentation result. The large language model is fine-tuned in this process.
[0034] It should be noted that the above method is based on an image segmentation network, which is constructed using a basic segmentation network. This basic segmentation network includes a pre-feature extraction module, a basic multi-scale module, and a segmentation module, and is trained using training images. Specifically, multiple trained basic multi-scale modules are concatenated according to multiple scales. The basic multi-scale module corresponding to the smallest scale is connected to the pre-feature extraction module, and the basic multi-scale module corresponding to the largest scale is connected to the segmentation module, thus obtaining the image segmentation network. The pre-feature extraction module is used to extract pre-features from the image at the smallest scale. The basic multi-scale module is used to obtain the input image at the current scale and the output features of the previous scale, and outputs the output features at the current scale. The segmentation module is used to perform image segmentation.
[0035] Specifically, please refer to Figure 2 , Figure 2This is a schematic diagram illustrating an application scenario of one embodiment of the image segmentation network of this application. For ease of understanding, the downsampling in this application is exemplified by a factor of 2. In specific scenarios, this can be customized, and this application does not impose specific limitations on this. The basic segmentation network includes a pre-feature extraction module, a basic multi-scale module, and a segmentation module. After feature extraction by the pre-feature extraction module, the training image yields scaled-down output features. The training image is then input into the basic multi-scale module to obtain image features. The basic multi-scale module groups the image features and the output features of the previous scale according to feature channels, obtaining multiple feature groups and determining the representative features corresponding to each feature group. Based on each channel of the image features and all representative features, the output features of the current scale are determined. The segmentation module is used to perform image segmentation based on the output features output by the basic multi-scale module. Therefore, the basic segmentation network only needs to be trained once during training.
[0036] Furthermore, for features of different scales required to be extracted in any scene, after determining all the scales required for the scene, multiple trained basic multi-scale modules can be connected in series according to multiple scales. The basic multi-scale module corresponding to the smallest scale is connected to the pre-feature extraction module, and the basic multi-scale module corresponding to the largest scale is connected to the segmentation module. In this way, an image segmentation network can be constructed. The basic multi-scale modules in the image segmentation network can be reused indefinitely, thereby improving the versatility of the image segmentation network in different scenes.
[0037] The above scheme acquires the original image at its maximum scale and multiple images at decreasing scales corresponding to the original image. Pre-feature extraction is performed on the image at the minimum scale to obtain the scaled-down output features. The images from the minimum to the maximum scale are used sequentially as input images, and the process iterates from the minimum to the maximum scale. The decreasing scale can be set according to the target size as needed. Image features corresponding to the input image at the current scale are acquired. These image features and the output features obtained from the previous scale are grouped according to feature channels, resulting in multiple feature groups. Each feature group includes features from some channels of the image features or output features. Features are compressed within multiple channels corresponding to the feature group to reduce computational requirements. Representative features corresponding to the feature group are determined. Based on each channel of the image features and all representative features, the output features at the current scale are determined so that the output features can fuse features from both scales. Furthermore, feature extraction and feature fusion can be performed in the same way for each two scales, making the method of acquiring output features at each scale universal. When the output features corresponding to the scale of the original image are obtained, image segmentation is performed using the final output features to obtain the image segmentation result, thereby enhancing the universality of image segmentation and reducing computational cost.
[0038] Please see Figure 3 , Figure 3This is a flowchart illustrating another embodiment of the image segmentation method of this application, the method comprising: S201: Obtain multiple images of decreasing scales corresponding to the original image at the largest scale, perform pre-feature extraction on the image at the smallest scale, and obtain the output features after scale reduction; wherein, traverse from the smallest scale to the largest scale, and use the image at each scale as the input image.
[0039] Specifically, the original image at the largest scale and multiple images at decreasing scales corresponding to the original image are obtained. The pre-feature extraction module is used to perform pre-feature extraction on the image at the smallest scale to obtain the output features after scale reduction.
[0040] It should be noted that pre-feature extraction of the smallest scale image to obtain the scaled output features includes: using the pre-feature extraction module to perform scale adjustment and feature extraction on the smallest scale image to obtain the scaled output features; wherein, the pre-feature extraction module obtains the features by truncating a portion of the network layers from the feature extraction network, with at least one network layer used for scale adjustment and multiple network layers used for feature extraction.
[0041] Specifically, the image at its smallest scale is input into the pre-feature extraction module. The network layers in the pre-feature extraction module perform scale adjustment and feature extraction on the image to obtain the scaled-down output features. The pre-feature extraction module obtains its output features by truncating a portion of the network layers from the feature extraction network. At least one network layer is used for scale adjustment, and multiple network layers are used for feature extraction. This simplifies the construction of the pre-feature extraction module while ensuring its effectiveness.
[0042] Alternatively, the feature extraction network may include: Visual Geometry Group (Vgg), ResNet residual network, or GoogleNet network. These pre-trained networks can be used to obtain pre-feature extraction modules by using parts of their structures.
[0043] For easier understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating an application scenario of one embodiment of the pre-feature extraction module of this application, using the Vgg16 network as an example for feature extraction. Figure 4 The network layers corresponding to the yellow blocks include convolutional layers, batch normalization (BN) layers, and activation layers for feature extraction. The network layers corresponding to the green blocks include pooling layers for downsampling. The input is a 3-channel RGB image. The output is The original Vgg16 network input image has a fixed size. However, since the structure after Vgg16 is removed here, there is no restriction on the size of the input image, as long as H and W of the input image are even numbers. The method is similar when using other networks; taking a downsampling factor of 2 as an example, it only requires truncating half the scale of the input portion of the network. However, when truncating a portion of the structure from these pre-trained networks as a pre-feature extraction module, the number of feature channels is determined by the feature extraction network. For example, after downsampling Vgg16 by half, the number of feature types output is 128. If the number of channels needs to be adjusted, the original number of channels in Vgg16 must be modified, then retrained, and finally, the required network layers must be truncated from the converged network as the pre-feature extraction module.
[0044] S202: Obtain the input image at the current scale, perform scale adjustment and feature extraction on the input image to obtain the image features corresponding to the input image; wherein, the image features have the same scale as the output features of the previous scale.
[0045] Specifically, the input image at the current scale is acquired, and the scale of the input image is adjusted and features are extracted to obtain the image features corresponding to the input image, so that the image features have the same scale as the output features at the previous scale, thereby ensuring the accuracy of feature extraction and ensuring scale uniformity.
[0046] In one embodiment, acquiring an input image at the current scale, performing scale adjustment and feature extraction on the input image to obtain image features corresponding to the input image includes: acquiring an input image at the current scale, performing multiple convolutions on the input image to obtain image features corresponding to the input image; wherein, the multiple convolutions include multiple single-step convolutions and at least one preset multi-step convolution, and single-step convolutions are included before and after the preset multi-step convolution.
[0047] Specifically, the input image at the current scale is acquired, and the input image is convolved multiple times. The convolution process includes multiple single-step convolutions and at least one preset multi-step convolution. The preset multi-step convolution includes single-step convolutions before and after the multi-step convolution, thereby improving the accuracy of feature extraction through multiple single-step convolutions and achieving controllable downsampling during feature extraction through the preset multi-step convolution.
[0048] It is understandable that after obtaining the input image at the current scale, the process of scaling and feature extraction of the input image at the current scale can be implemented through network layers. When the output features of each scale are output by the basic multi-scale module, the image features are implemented by the network layers in the basic multi-scale module, which includes multiple convolutional network layers, dual-scale fusion network layers, and upsampling network layers connected in sequence.
[0049] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an application scenario of one implementation method of the basic multi-scale module of this application, wherein, Figure 5 The leftmost element is the input to the basic multi-scale module. The input is either the downsampled image or the original image, and the format of the input image is as follows: The input to the dual-scale fusion network layer in the middle part of the basic multi-scale module is the image features of the input image and the output features of the previous scale, in the following format: Here, H and W are based on the current scale. The feature format output by the dual-scale fusion network layer remains unchanged. The dual-scale fusion network layer is followed by an upsampling network layer, which amplifies the data in scale. The upsampling method used can be bilinear upsampling, for example... The features were obtained after bilinear upsampling. The upsampling method can also employ nearest neighbor or bicubic methods. Therefore, the final output feature format of the basic multi-scale module is as follows: .
[0050] For ease of understanding, we'll use a downsampling factor of 2 as an example. The yellow rectangles represent convolutional, BN, and activation layers with a stride of 1, while the red rectangles represent convolutional, BN, and activation layers with a stride of 2. Therefore, the convolutional layer in the red rectangle inherently includes a downsampling operation. The ellipsis indicates more yellow rectangles. Set the appropriate number of blocks according to your actual needs, and you can also set the number of red rectangles according to the downsampling factor requirements.
[0051] In a specific implementation scenario, after three single-stride convolutions, a pre-defined multi-stride convolution is connected, followed by two more single-stride convolutions. This corresponds to... Figure 5 In this configuration, three yellow blocks are followed by one red block, and then two more yellow blocks, ensuring the accuracy of feature extraction and meeting the requirements of image segmentation. This configuration can be adjusted according to different implementation scenarios; this application does not impose specific restrictions on it.
[0052] S203: Based on the total number of channels corresponding to the image features and the preset number of grouped channels, the image features are grouped. Based on the total number of channels of the output features of the previous scale and the preset number of grouped channels, the output features are grouped to obtain multiple feature groups.
[0053] Specifically, the image features are grouped based on the total number of feature channels included in the image features and the preset number of grouped channels. The output features are grouped based on the total number of feature channels included in the output features of the previous scale and the preset number of grouped channels, resulting in multiple feature groups.
[0054] Understandably, the preset number of grouped channels can evenly divide the total number of channels, thereby dividing all channels of the image features into multiple feature groups, and dividing all channels of the output features of the previous scale into multiple feature groups, ensuring the uniformity and accuracy of the feature groups.
[0055] It should be noted that the number of channels can be adjusted according to actual needs. Specifically, modifying the number of channels in the output features of the previous convolutional layer only requires configuring the convolutional weight parameters of the previous layer. However, modifying the number of channels in the pre-extracted features or the output features of the previous level's basic multi-scale module requires modifying the output channels of both the pre-feature extraction module and the basic multi-scale module.
[0056] For ease of explanation, this application sets the number of channels for both features to 128, and assumes the preset grouping channel size is C. In deep learning, the number of channels can be represented as the number of feature types. For example... The data has 128 channels, which means it contains 128 types of features. Therefore, there are a total of [number missing] features. Grouping. Grouping can be done by grouping C consecutive adjacent channels into one group. The size of C should be set to be divisible by 256, and preferably also satisfy C being a power of 2, such as C being 2, 4, 8, 16, and 32, etc., where C is generally set to 16 or 32.
[0057] S204: Perform feature extraction on each feature group to obtain the representative features corresponding to each feature group.
[0058] Specifically, feature extraction is performed on each feature group to obtain representative features corresponding to each image group.
[0059] It is understandable that the data size of each group can be represented as This data is convolved, then processed by a batch normalization (BN) layer and an activation layer to obtain the data representation. This data represents the representative features corresponding to the grouping. If there are 1 group, then a total of 100 groups can be obtained. One representative feature.
[0060] S205: Concatenate all representative features to obtain representative combined features. Concatenate each channel of the image features with the representative combined features to obtain the interleaved combined features corresponding to each channel.
[0061] Specifically, all representative features are concatenated to obtain representative combined features, thereby compressing the two input features into representative combined features. Each channel of the image features is concatenated with the representative combined features to obtain the interleaved combined features corresponding to each channel. This ensures that the interleaved combined features include features of the two scales corresponding to the two levels, and can relatively emphasize the current scale.
[0062] Please see Figure 6 , Figure 6 This is a schematic diagram of an application scenario of one embodiment of the dual-scale fusion network layer of this application. In the dual-scale fusion network layer, the image features and the output features of the previous scale are grouped, the representative features corresponding to the feature groups are extracted, the representative features are connected to form representative combination features, each channel of the image features is concatenated with the representative combination features to obtain the interleaved combination features corresponding to each channel, and feature extraction can be performed on the interleaved combination features after obtaining them.
[0063] It is understandable that the process of grouping image features and output features from the previous scale, and finally obtaining interleaved combined features, is implemented in the dual-scale fusion network layer.
[0064] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an application scenario of one embodiment of obtaining interleaved combination features according to this application. Each channel in the image features is concatenated with a representative combination feature to obtain interleaved combination features. When the image features have a total of 128 channels, 128 interleaved combination features can be obtained. Each interleaved combination feature includes... One channel.
[0065] S206: Fuse the interleaved combination features corresponding to all channels to obtain dual-scale fused features, and upsample the dual-scale fused features to obtain the output features at the current scale.
[0066] Specifically, the interleaved combination features corresponding to all channels are fused to obtain dual-scale fusion features, so that the dual-scale fusion features can integrate all channels of image features and representative combination features, ensuring the fusion degree of features at different scales.
[0067] Furthermore, the dual-scale fusion features are upsampled to obtain the output features at the current scale, thereby ensuring that the output features at the current scale are compatible with the current scale.
[0068] In one embodiment, the interleaved combination features corresponding to all channels are fused to obtain dual-scale fused features, including: extracting features from each interleaved combination feature to obtain single-channel combination features corresponding to each channel, and concatenating the single-channel combination features of all channels to obtain dual-scale fused features.
[0069] Specifically, feature extraction is performed separately for each interleaved combination feature to obtain the single-channel combination feature corresponding to each channel, so that the single-channel combination feature can fully integrate the individual channels of the image features and the representative combination features. The single-channel combination features of all channels are spliced together to obtain the dual-scale fusion feature, ensuring that the features of the two scales corresponding to the two levels can be fully integrated and can relatively emphasize the current scale.
[0070] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating an application scenario for obtaining dual-scale fusion features according to one implementation method of this application. Convolutional, batch normalization (BN) layers, and activation layers are independently applied to each interleaved feature to obtain... The single-channel combination features. A total of 128 can be obtained. The scale of single-channel combined features is then concatenated to obtain... The dual-scale fusion feature.
[0071] It should be noted that if the standard is used... Depthwise convolution combines two... The data is fused to obtain 1 The data, then a total of The multiplication operation is required. However, using the method in this application, the computation mainly focuses on representative feature extraction and staggered combination feature extraction. The representative feature extraction stage requires... The multiplication operation is required in the interleaved combination feature extraction stage. The number of multiplication operations. Therefore, the improvement factor of the multiplication logic in this application is: When C is 16, the improvement factor is 13. When C is 32, the improvement factor is 23. Therefore, the above design can save computation and thus conserve computing power, and can achieve the fusion of various feature types. In the fusion process, representative features of feature groups are first constructed and combined to obtain representative combined features. The representative combined features and each channel of the image features are interleaved to obtain interleaved combined features. Convolution is performed on each interleaved combined feature, and the convolution result uses the information of all representative features, ultimately achieving the fusion of feature types.
[0072] S207: In response to obtaining the scale-corresponding output features of the original image, perform image segmentation using the final output features.
[0073] Specifically, when the output features corresponding to the scale of the original image are obtained, the final output features are used to perform image segmentation to obtain the image segmentation result, thereby enhancing the versatility of image segmentation and reducing computational power.
[0074] It is understandable that the above method is based on an image segmentation network, which is constructed using a base segmentation network. This base segmentation network includes a pre-feature extraction module, a base multi-scale module, and a segmentation module, and is trained using training images. Multiple trained base multi-scale modules are concatenated according to multiple scales. The base multi-scale module corresponding to the smallest scale is connected to the pre-feature extraction module, and the base multi-scale module corresponding to the largest scale is connected to the segmentation module, thus obtaining the image segmentation network. The pre-feature extraction module is used to extract pre-features from the image at the smallest scale. The base multi-scale module is used to obtain the input image at the current scale and the output features of the previous scale, and outputs the output features at the current scale. The segmentation module is used to perform image segmentation. Notably, the base multi-scale module in the image segmentation network can be reused indefinitely, thereby improving the versatility of the image segmentation network in different scenarios.
[0075] It should be noted that the basic multi-scale module includes multiple convolutional network layers, a dual-scale fusion network layer, and an upsampling network layer connected in sequence. The input image at the current scale is input to the first convolutional network layer. At least one of the multiple convolutional network layers performs downsampling. The convolutional network layer before the dual-scale fusion network layer outputs image features with the same output feature scale as the previous scale. The output features of the previous scale are input to the dual-scale fusion network layer. The number of feature channels of the image features is greater than or equal to the number of feature channels of the output features of the previous scale. Representative features are obtained in the dual-scale fusion network layer. The upsampling network layer outputs output features that match the current scale.
[0076] Specifically, please refer to [the relevant document] again. Figure 5 The basic multi-scale module includes multiple convolutional network layers, a dual-scale fusion network layer, and an upsampling network layer connected in sequence. The input image at the current scale is input to the first convolutional network layer. At least one of the multiple convolutional network layers performs downsampling. The features output by the convolutional network layer connected to the dual-scale fusion network layer are input to the dual-scale fusion network layer. This convolutional network layer outputs image features with the same scale as the output features of the previous scale. The number of feature channels of the image features is greater than or equal to the number of feature channels of the output features of the previous scale, so that the features corresponding to the current scale can be extracted as finely as possible within each current scale. Representative features can be obtained in the dual-scale fusion network layer, thereby reducing computational power. The upsampling network layer outputs output features that match the current scale, thereby ensuring scale consistency.
[0077] It should be noted that the image downsampling in the above embodiments of this application uses a factor of 2 each time, but in practice it can be adjusted as needed, for example, using a factor of 4. Then the network layer corresponding to the pre-feature extraction module will also extract the corresponding portion.
[0078] Please see Figure 9 , Figure 9 This is a schematic diagram of another implementation of the pre-feature extraction module of this application. The pre-feature extraction module includes two pooling layers to achieve downsampling by 4 times. In other scenarios, it can be truncated as needed, which will not be described in detail here.
[0079] Please see Figure 10 , Figure 10 This is a schematic diagram of another implementation of the basic multi-scale module of this application. The red rectangular blocks represent two convolutional, BN, and activation layers with a stride of 2, thereby achieving a downsampling of 4 times. In other scenarios, these can be set as needed, which will not be elaborated on in this application.
[0080] It is understandable that the basic segmentation network mentioned above only needs to be trained once to be used to build an image segmentation network. Furthermore, the basic multi-scale module in the basic segmentation network can be reused indefinitely, improving the versatility of the image segmentation network. In addition, the basic multi-scale module can effectively reduce computing power.
[0081] Please see Figure 11 , Figure 11 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other. The memory 301 stores program data (not shown in the figure). The processor 302 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0082] Please see Figure 12 , Figure 12 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.
[0083] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0086] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.
Claims
1. An image segmentation method, characterized in that, The method includes: The process involves obtaining multiple images at decreasing scales corresponding to the original image at the largest scale, performing pre-feature extraction on the image at the smallest scale, and obtaining the output features after scale reduction. Specifically, the process iterates from the smallest scale to the largest scale, with each scale image serving as the input image. The image features corresponding to the input image at the current scale are obtained. The image features and the output features at the previous scale are grouped according to the feature channels to obtain multiple feature groups and the representative features corresponding to the feature groups are determined. Based on each channel of the image features and all the representative features, the output features at the current scale are determined. Each feature group includes features from some channels of the image features or output features. In response to obtaining the scale-corresponding output features of the original image, image segmentation is performed using the final output features.
2. The image segmentation method according to claim 1, characterized in that, The process of obtaining image features corresponding to the input image at the current scale, grouping the image features and the output features at the previous scale according to feature channels to obtain multiple feature groups, and determining the representative features corresponding to the feature groups includes: The input image at the current scale is acquired, and the scale of the input image is adjusted and features are extracted to obtain the image features corresponding to the input image; wherein, the image features have the same scale as the output features at the previous scale. Based on the total number of channels corresponding to the image features and the preset number of grouped channels, the image features are grouped. Based on the total number of channels of the output features of the previous scale and the preset number of grouped channels, the output features are grouped to obtain multiple feature groups. Feature extraction is performed on each of the feature groups to obtain the representative feature corresponding to each feature group.
3. The image segmentation method according to claim 2, characterized in that, The process of acquiring the input image at the current scale, performing scale adjustment and feature extraction on the input image to obtain the image features corresponding to the input image includes: The input image at the current scale is obtained, and the input image is convolved multiple times to obtain the image features corresponding to the input image; wherein, the multiple convolutions include multiple single-step convolutions and at least one preset multi-step convolution, and the single-step convolution is included before and after the preset multi-step convolution.
4. The image segmentation method according to claim 1, characterized in that, The process of determining the output features at the current scale based on each channel of the image features and all the representative features includes: All the representative features are concatenated to obtain representative combined features. Each channel of the image feature is concatenated with the representative combined features to obtain the interleaved combined features corresponding to each channel. The interleaved features corresponding to all channels are fused to obtain dual-scale fused features. The dual-scale fused features are then upsampled to obtain the output features at the current scale.
5. The image segmentation method according to claim 4, characterized in that, The process of fusing the interleaved combination features corresponding to all channels to obtain dual-scale fused features includes: Feature extraction is performed on each of the interleaved combination features to obtain the single-channel combination features corresponding to each channel. The single-channel combination features of all channels are then concatenated to obtain the dual-scale fusion feature.
6. The image segmentation method according to claim 1, characterized in that, The process of pre-feature extraction from the smallest-scale image to obtain scaled-down output features includes: The pre-feature extraction module is used to scale and extract features from the smallest scale image to obtain the output features after scale reduction; wherein, the pre-feature extraction module is obtained by truncating a portion of the network layers from the feature extraction network, at least one network layer is used for scale adjustment and multiple network layers are used for feature extraction.
7. The image segmentation method according to any one of claims 1-6, characterized in that, The method is based on an image segmentation network, which is constructed using a basic segmentation network. The basic segmentation network includes a pre-feature extraction module, a basic multi-scale module, and a segmentation module. The basic segmentation network is trained using training images. The image segmentation network is obtained by concatenating multiple trained basic multi-scale modules according to multiple scales. The basic multi-scale module corresponding to the smallest scale is connected to the pre-feature extraction module, and the basic multi-scale module corresponding to the largest scale is connected to the segmentation module. The pre-feature extraction module is used to perform pre-feature extraction on the image at the smallest scale. The basic multi-scale module is used to obtain the input image at the current scale and the output features at the previous scale, and output the output features at the current scale. The segmentation module is used to perform image segmentation.
8. The image segmentation method according to claim 7, characterized in that, The basic multi-scale module includes multiple convolutional network layers, a dual-scale fusion network layer, and an upsampling network layer connected in sequence. In this process, the input image at the current scale is input to the first convolutional network layer. At least one of the multiple convolutional network layers performs downsampling. The convolutional network layer before the dual-scale fusion network layer outputs image features with the same output feature scale as the previous scale. The output features of the previous scale are input to the dual-scale fusion network layer. The number of feature channels of the image features is greater than or equal to the number of feature channels of the output features of the previous scale. The representative features are obtained in the dual-scale fusion network layer. The upsampling network layer outputs output features that match the current scale.
9. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-8.
10. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, it implements the method as described in any one of claims 1-8.