Image segmentation method, terminal equipment and storage medium

Abstract

The invention belongs to the technical field of image processing, and provides an image segmentation method, terminal equipment and a storage medium, and the method comprises the steps: carrying out the edge extraction of an input image, and obtaining the edge features of the input image; processing the input image to obtain an image block embedding vector; segmenting the input image into n image block sequences with different resolutions, and obtaining the feature representation of each image block sequence; fusing the feature representation of the (i + 1) th image block sequence with the output feature of the ith encoder through a step-by-step encoding mode to obtain an encoding feature; decoding the coding features to obtain decoding features; fusing and adjusting the decoding features and the edge features to obtain adjusting features; performing semantic segmentation on the adjustment features; according to the image segmentation method provided by the invention, the output detail information is increased, and the good effect of eye image segmentation is ensured.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image segmentation method, a terminal device and a storage medium. Background Art

[0002] The deep learning-based eye image segmentation method can provide clinicians with quantitative or qualitative feedback, such as the possibility of disease, related physiological structures, and the shape of lesions, thereby improving the efficiency of doctors' diagnosis and reducing costs, which is of great significance for protecting eye health. However, due to the influence of factors such as light and cataracts, fundus images usually have low contrast and weak edge information. The boundary between the optic cup and the optic disc is not obvious and contains many tiny blood vessels with a length ranging from 1 to 10 pixels. The edge information of detailed structures such as tiny blood vessels is easily lost in downsampling operations and is difficult to recover through upsampling operations.

[0003] Therefore, how to enhance the edge feature extraction capability of neural networks is a difficult problem in eye image segmentation. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an image segmentation method, a terminal device and a storage medium to solve the problem that the existing neural network has insufficient edge feature extraction capabilities, resulting in poor eye image segmentation effects.

[0005] A first aspect of an embodiment of the present application provides an image segmentation method, comprising:

[0006] Performing edge extraction on an input image to obtain edge features of the input image;

[0007] Processing the input image to obtain an image block embedding vector;

[0008] Segment the input image into n image block sequences of different resolutions, and obtain a feature representation of each image block sequence;

[0009] By means of step-by-step encoding, the feature representation of the (i+1)th image block sequence is fused with the output feature of the i-th encoder to obtain a coding feature;

[0010] Decoding the encoded features to obtain decoded features;

[0011] The decoding feature and the edge feature are fused and adjusted to obtain an adjustment feature;

[0012] performing semantic segmentation on the adjustment features;

[0013] Outputting the image obtained by the semantic segmentation;

[0014] Wherein, i=1, 2, ..., n-1, n is an integer greater than 1, and the feature representation of the first image block sequence is fused with the image block embedding vector.

[0015] The second aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the image segmentation method provided in the first aspect of the embodiment of the present application are implemented.

[0016] A third aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the image segmentation method provided in the first aspect of the embodiments of the present application are implemented.

[0017] The first aspect of the embodiment of the present application provides an image segmentation method, which extracts the edge of an input image to obtain the edge features of the input image; processes the input image to obtain an image block embedding vector; segments the input image into n image block sequences of different resolutions, and obtains a feature representation of each image block sequence; fuses the feature representation of the i+1th image block sequence with the output feature of the i-th encoder in a step-by-step encoding manner to obtain an encoding feature; decodes the encoding feature to obtain a decoding feature; fuses and adjusts the decoding feature with the edge feature to obtain an adjustment feature; performs semantic segmentation on the adjustment feature; and outputs the image obtained by the semantic segmentation; wherein i=1,2,…,n-1, n is an integer greater than 1, the feature representation of the first image block sequence is fused with the image block embedding vector, and the present application enhances weak boundary information through multi-scale local area input, and uses the extracted detail features (i.e., edge features) to adjust the extracted global features (i.e., decoding features), thereby increasing the output detail information and ensuring a good effect of eye image segmentation.

[0018] It can be understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of an image segmentation method provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of an image segmentation method provided by another embodiment of the present application;

[0022] Figure 3 is a flowchart of an image segmentation method provided by another embodiment of the present application;

[0023] Figure 4 is a flowchart of an image segmentation method provided by another embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the structure of an image segmentation device provided in an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the term “and / or” used in the specification and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] The image segmentation method provided in the first aspect of the embodiment of the present application can be executed by the processor of the terminal device when running a computer program with corresponding functions, by extracting the edge of the input image to obtain the edge feature of the input image; processing the input image to obtain an image block embedding vector; segmenting the input image into n image block sequences of different resolutions, and obtaining the feature representation of each image block sequence; by encoding in a step-by-step manner, fusing the feature representation of the i+1th image block sequence with the output feature of the i-th encoder to obtain the encoding feature; decoding the encoding feature to obtain the decoding feature; fusing and adjusting the decoding feature with the edge feature to obtain the adjustment feature; performing semantic segmentation on the adjustment feature; and outputting the image obtained by the semantic segmentation; wherein i=1,2,…,n-1, n is an integer greater than 1, the feature representation of the first image block sequence is fused with the image block embedding vector, and the present application enhances weak boundary information through multi-scale local area input, and uses the extracted detail features (i.e., edge features) to adjust the extracted global features (i.e., decoding features), thereby increasing the output detail information, thereby ensuring a good effect of eye image segmentation.

[0033] In the application, the input image may be a color fundus image obtained by an ophthalmic imaging device having at least one of the functions of color fundus photography, optical coherence tomography, fluorescein angiography, indocyanine green angiography, autofluorescence imaging, wide-field fundus imaging, etc., or other types of fundus scanning images. The terminal device may be a computing device having at least one of the functions of image classification, image recognition, machine vision, etc., such as a robot, a mobile phone, a camera, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a tablet computer, a laptop computer, a personal computer (PC), a server, a dedicated medical device, etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0034] like Figure 1 As shown, the image segmentation method provided in the embodiment of the present application includes the following steps S101 to S108:

[0035] Step S101: extract edges of an input image to obtain edge features of the input image.

[0036] In the application, the image data may be obtained from the local storage space of the terminal device, or the storage space of any other device (eg, an ophthalmic imaging device) communicating with the terminal device, for example, obtaining an input image from the ophthalmic imaging device.

[0037] In applications, traditional edge detection operators (such as Canny and Sobel operators) can be used to extract edge features of images, or deep learning methods (such as edge detection networks based on convolutional neural networks) can be used to generate edge features F. t Edge features can enhance the sensitivity to the boundaries of objects in an image, which is especially important in the detection of small structures such as blood vessels. It helps to preserve the edge information of detailed structures such as tiny blood vessels and reduce information loss in subsequent processing.

[0038] Step S102: Process the input image to obtain an image block embedding vector.

[0039] In the application, the image patch embedding vector is used as the input of the entire encoder. The feature representation of the input image can be extracted using models such as convolutional neural networks (CNN) or visual transformers (ViT). In the visual transformer, the image is divided into multiple fixed-size patches, each of which is flattened and mapped to an embedding vector, the image patch embedding vector. The image patch embedding vector can capture the information of different positions in the image and preserve the spatial position relationship. The global dependencies between image patches can be better learned through models such as visual transformers.

[0040] Step S103: Segment the input image into n image block sequences of different resolutions, and obtain a feature representation of each image block sequence.

[0041] In applications, downsampling operations (such as max pooling or convolutional layers with a stride of 2) can be used to create image patch sequences of different resolutions. A convolutional layer is applied to the image patch sequence of each resolution to extract feature representations, and feature representations of each image patch sequence are obtained. Multi-scale feature extraction helps capture structures of different sizes in the image, and feature maps of different resolutions can help the model understand different levels of information from global to local.

[0042] Step S104: By means of a step-by-step encoding method, the feature representation of the (i+1)th image block sequence is fused with the output feature of the i-th encoder to obtain an encoding feature.

[0043] In applications, progressive encoding refers to the process in which the feature map is gradually compressed (resolution is reduced) as the depth of the network increases, while the feature representation becomes more abstract. In this process, each layer of the encoder performs some form of feature extraction and downsampling operations on the input in order to capture higher-level abstract features. For example, the input image is first processed to obtain an image block embedding vector, and then a multi-level encoder architecture is used. In the first encoder, the feature representation of the first image block sequence is fused with the image block embedding vector, and then input into the next encoder, and so on. The feature map of the current level is fused with the feature map of the previous higher resolution level using a recursive structure until the required minimum resolution feature representation is reached, that is, i takes values ​​of 1, 2, ..., n-1 in sequence until the nth (n is an integer greater than 1) encoder is output. Finally, the feature representations of all levels are fused together to form a comprehensive feature representation containing multi-scale information.

[0044] Step S105: Decode the encoded feature to obtain a decoded feature.

[0045] In the application, the decoder structure is symmetrical to the encoder structure. The encoder and decoder can fuse information of different scales through jump links and restore the spatial information loss caused by the regional merging layer. In the decoding process, upsampling operations (such as transposed convolution or nearest neighbor interpolation) are used to gradually restore the resolution of the feature map, and the feature map in the decoder is fused with the feature map in the encoder through jump links. The decoding process helps to reconstruct the details of the original image, and combining the features of the encoder and decoder helps to restore the lost detail information.

[0046] Step S106: Fusing and adjusting the decoding feature and the edge feature to obtain an adjusted feature.

[0047] In the application, the decoding features can be fused with the edge features by point-to-point multiplication or addition, and the convolution layer can be used for feature adjustment to ensure that the fused feature map is consistent with the size of the final output. Among them, the addition of edge features helps to enhance the boundary information of structures such as blood vessels, and feature adjustment ensures the coherence and consistency of the output. Through this step, the global information and local detail information can be further enhanced. The decoding features are directly convolved through the original image, so they have little effect on the detail information. The information obtained by the Transformer (i.e., the decoding features) will lose some information about local microvessels. The edge features obtained by convolution of the original image are used to fine-tune the decoding features obtained using the Transformer, thereby enhancing the model's ability to recognize microvessels.

[0048] Step S107: performing semantic segmentation on the adjustment features.

[0049] In applications, the adjusted features can be trained with a cross-entropy loss function to achieve semantic segmentation of blood vessels. This process involves multiple steps, including feature extraction, feature adjustment, loss calculation, and the final segmentation output. By carefully designing these steps, the accuracy of blood vessel segmentation can be significantly improved, thereby providing valuable auxiliary information to clinicians. In addition, the use of feature adjustment and cross-entropy loss function helps the model better understand and process the detailed information in the image, which is crucial for the segmentation of small structures such as blood vessels. In addition to using the cross entropy loss function for training to achieve semantic segmentation of blood vessels, other loss functions can also be used, such as Dice loss function, Focal loss function, Tversky loss function, IoU loss function, Jaccard loss function, BCE-Dice combined loss function, BCE-IoU combined loss function, Mixed loss function, Wasserstein loss function, L1 loss function, mean absolute error loss function, etc. Semantic segmentation can assign a specific category label to each pixel in the image, which helps to clearly distinguish different types of structures (such as blood vessels, optic discs, optic cups, etc.).

[0050] Step S107: output the image obtained by the semantic segmentation.

[0051] In the application, the final segmentation result is output, that is, an image with a category label assigned to each pixel. The segmentation result can be further optimized using threshold processing or post-processing techniques. The output segmentation map can intuitively show the location and shape of different structures, and the segmentation result can provide valuable reference information for clinicians.

[0052] In one embodiment, step S101 includes:

[0053] The input image is input into one or more convolutional layers for edge extraction to obtain edge features (low-level features) of the input image.

[0054] In the application, the input image can be fed into one or more convolutional layers (such as 1, 2, 3, etc.) for edge extraction, which is used to capture low-level visual features such as blood vessel edges, textures, and basic shapes from the original eye map image. By using filters, the convolutional layer can detect local changes in the eye map and extract low-level feature maps representing blood vessel edge features. The filter can identify the edges of blood vessels in different directions and scales, thereby generating a set of multi-channel feature maps, each channel containing a specific type of blood vessel edge information, that is, the edge features (low-level features) of the input image. These blood vessel edge features form the basis of subsequent processing steps and can be used to adjust the decoded features to ultimately achieve accurate eye map blood vessel segmentation.

[0055] In one embodiment, step S102 includes the following steps S201 and S202:

[0056] Step S201: Divide the input image into multiple image blocks.

[0057] In the application, input the image First, it is divided into a series of non-repeating regional fragments (i.e., image blocks) of size 4×4 pixels. Among them, H refers to the number of pixels of the image in the vertical direction, i.e., the height; W refers to the number of pixels of the image in the horizontal direction, i.e., the width; C refers to the number of color information in the image, i.e., the number of channels. For grayscale images, the number of channels is 1, and for color images (such as RGB images), the number of channels is usually 3 (three channels: red, green, and blue). This division method ensures that each fragment is independent and there is no overlap in the entire image. Choosing a size of 4×4 pixels can effectively balance computational complexity and local feature capture capabilities. For example, if the size of the original image is H×W pixels, then the number of fragment sequences after division will be H / 4×W / 4, which helps to maintain the spatial structure information of the image in subsequent processing. Image block division methods of other sizes can also be used, such as m×m, m=2 k , k ranges from 1≤k≤8. Then the output feature embedding size will be adjusted to H / m×W / m accordingly to ensure that the density of feature embedding matches the size of the image block.

[0058] Step S202: Perform linear transformation processing on the multiple image blocks to obtain image block embedding vectors.

[0059] In applications, these 4×4 (or m×m) pixel image blocks are transformed into higher-dimensional feature vectors, namely, image block embedding vectors (Patch Embedding). Each image block is mapped to a C-dimensional space through one or more linear layers, where C represents the dimension of the feature vector. This transformation helps capture complex patterns within the image block and provides richer representations for subsequent deep learning models.

[0060] In one embodiment, step S103 includes the following steps S301 and S302:

[0061] Step S301: Segment the input image into n image block sequences with different resolutions.

[0062] In the application, the input image can be divided into a sequence of image blocks with different resolutions, such as: The input image can also be divided into three image block sequences with a resolution of In addition to the above segmentation forms, n can also be set to 2, 5, 6, etc. to obtain image block sequences with higher resolutions. These image block sequences with different resolutions help the model capture information at different scales, from local details to global structures. In this way, the model can effectively learn multi-level features in the image.

[0063] Step S302: perform linear transformation and normalization processing on the n image block sequences respectively to obtain feature representations of the n image block sequences.

[0064] In the application, the first image block sequence is linearly transformed and normalized (such as passing through a linear layer and a normalization layer) to obtain a feature representation of the first image block sequence, the second image block sequence is linearly transformed and normalized to obtain a feature representation of the second image block sequence, and so on, the nth image block sequence is linearly transformed and normalized to obtain a feature representation of the nth image block sequence, thereby obtaining a feature representation of n image block sequences.

[0065] In the application, the image block sequence is linearly transformed and normalized to extract meaningful feature representations from the input image block sequence. Specifically, the original pixel values ​​of each image block are first converted into a higher-dimensional feature vector through a linear transformation (usually through a linear layer, such as a fully connected layer). This process helps capture the complex patterns and structures in the image blocks. Subsequently, these feature vectors are standardized through normalization operations (such as batch normalization or instance normalization) to ensure that each feature dimension has the same numerical range, which helps to improve the training stability and convergence speed of the model. After these two steps, each image block is converted into a feature vector in a unified format. These feature vectors together constitute the feature representation of the image block sequence, providing a richer and more standardized input for subsequent deep learning models.

[0066] In one embodiment, Figure 2 As shown, step S104 includes the following steps S401 and S406:

[0067] Step S401: After the image block embedding vector is input into the first encoder for encoding, it is linked with the first image block sequence to obtain a first down-sampling link feature.

[0068] In applications, such as Figure 3 As shown in the figure, when m is 4, n is 3, and C is 3, the image block embedding vector obtained after segmentation of the original image is input into the encoder 1 for encoding, that is, it is processed by two layers of Swin-Transformer modules. The Swin-Transformer module uses the window self-attention mechanism to capture the dependency between features, and transforms the features through self-attention and feedforward networks, that is, feature encoding. Then the encoded features are linked with the first image block sequence by concatenation or element-wise addition to obtain the first down-sampled link feature. Among them, the feature representation of the first image block sequence refers to (resolution is The 48-dimensional feature representation of each image block in the image blocks.

[0069] Step S402: Perform a linear transformation on the first down-sampled link feature to obtain an output feature of the first encoder.

[0070] In applications, the concatenated features are adjusted through a linear transformation (e.g., through a linear layer) to adjust the dimension of the features.

[0071] Step S403: down-sample the output features of the i-th encoder.

[0072] In the application, the output features of the first encoder and the output features of the second encoder (when n=3, the maximum value of i in this step is 2) are downsampled. The size of the feature map is reduced by downsampling operations (such as stride convolution or region merging layers), thereby reducing computational costs and extracting more abstract features. Downsampling also helps the model build a hierarchical feature representation to capture information at different scales.

[0073] Step S404: Input the result of the downsampling process to the (i+1)th encoder for encoding, and then link it with the feature representation of the input (i+1)th image block sequence to obtain the (i+1)th downsampling link feature.

[0074] In the application, the output features of the first encoder are downsampled (such as the region merging layer) and then input to the second encoder for encoding, and then linked with the feature representation of the second image block sequence input to obtain the second downsampled linked features. The output features of the second encoder are downsampled and then input to the third encoder for encoding, and then linked with the feature representation of the third image block sequence input to obtain the third downsampled linked features. The feature representation of the second image block sequence refers to (resolution is ) image blocks. The feature representation of the third image block sequence is (resolution is ) image blocks, a 256-dimensional feature representation of each image block.

[0075] Step S405: Perform a linear transformation on the i+1th down-sampled link feature to obtain the i+1th output feature of the encoder.

[0076] In the application, the second down-sampled link feature is subjected to a linear transformation process (such as through a linear layer) to obtain the output feature of the second encoder. The third down-sampled link feature is subjected to a linear transformation process to obtain the output feature of the third encoder.

[0077] Step S406: After downsampling the output features of the nth encoder, the features are input into one or more sliding window self-attention modules for fusion to obtain encoding features.

[0078] In the application, the output features of the nth encoder (when n=3, the output features of the third encoder) are down-sampled and input into one or more (such as 1, 2, 3) sliding window self-attention modules (Swin-Transformer modules) for fusion to obtain the encoding features, that is, the features after global and local fusion.

[0079] In one embodiment, step S105 includes the following step S501:

[0080] S501. By means of step-by-step decoding, the output features of the ni-th encoder and the output features of the ith decoder are merged to obtain decoding features.

[0081] In the application, the output features of the ni-th encoder are feature representations processed by multi-level encoders. These features contain higher-level abstract information, but have lower resolution. The output features of the ith decoder are feature representations that are gradually upsampled after being processed by the decoder. These features have higher resolution than the output features of the encoder. In the decoding stage, the output features of the decoder are increased in resolution through upsampling operations (such as transposed convolution, bilinear interpolation, region expansion, etc.) to match the size of the feature map of the encoder. In the process of decoding step by step, the output features of the ith decoder are upsampled and fused step by step with the output features of the ni-th encoder, such as skip connections, to finally obtain the decoded features. The decoded features combine the abstract information of the encoder and the detailed information gradually restored by the decoder, and obtain a richer feature representation. This process helps to gradually restore the resolution of the feature map and integrate information from different levels. The decoded features finally obtained contain a more comprehensive feature representation, providing a better foundation for subsequent image segmentation.

[0082] In one embodiment, Figure 4 As shown, step S501 includes the following steps S5011 to S5016:

[0083] Step S5011: perform a first upsampling process on the encoded feature, and perform a jump link with the output feature of the nth encoder to obtain a first upsampling link feature.

[0084] In the application, the encoded features are first upsampled, such as by increasing the resolution by a region expansion layer, which can be 2 or 4 times. The upsampling operation can also be completed by using any of the transposed convolutional layers, nearest neighbor interpolation, or bilinear interpolation. The upsampled features are fused with the output features of the deepest (nth) encoder through a skip link to obtain the first upsampled link feature.

[0085] Step S5012: input the first up-sampled link feature into the first decoder for decoding to obtain the output feature of the first decoder.

[0086] In the application, the decoder includes a linear layer and two Swin-Transformer modules. The first upsampled link feature is fed into the first decoder, and after being processed by a linear layer and two Swin-Transformer modules, the output feature of the first decoder is obtained.

[0087] Step S5013: Perform a first upsampling process on the output features of the i-th decoder.

[0088] In the application, the first upsampling process is performed on the output features of the first decoder, and the first upsampling process is performed on the output features of the second decoder. The first upsampling process can be performed by expanding the area expansion layer by 2 times, 4 times, etc.

[0089] Step S5014: perform jump linking on the result of the first upsampling process and the output feature of the ni-th encoder to obtain the (i+1)-th upsampling link feature.

[0090] In the application, the result of the first upsampling process on the output feature of the first decoder is jump-linked with the output feature of the second encoder to obtain the second upsampling link feature. The result of the first upsampling process on the output feature of the second decoder is jump-linked with the output feature of the first encoder to obtain the third upsampling link feature.

[0091] Step S5015: input the i+1th up-sampled link feature to the i+1th decoder for decoding to obtain the output feature of the i+1th decoder.

[0092] In the application, the second upsampled link feature is input to the second decoder for decoding (processed by a linear layer and two Swin-Transformer modules), and the output feature of the second decoder is obtained. The third upsampled link feature is input to the third decoder for decoding (processed by a linear layer and two Swin-Transformer modules), and the output feature of the third decoder is obtained.

[0093] Step S5016: Perform a second upsampling process on the output features of the nth decoder to obtain the decoded features.

[0094] In the application, the output features of the third decoder are subjected to a second upsampling process, such as region expansion, with a multiple of 4 (other multiples can be selected as needed), to obtain the decoded features F c .

[0095] In one embodiment, step S106 includes:

[0096] Using the formula fusing and adjusting the decoding feature and the edge feature to obtain an adjusted feature;

[0097] in, represents pixel-by-pixel multiplication, represents pixel-by-pixel addition, F c is the decoding feature, F t is the edge feature, F m To adjust the characteristics.

[0098] In the application, two convolutional layers are first used to obtain the edge features (low-level features) F of the input image. t , and then use these edge features F t Adjust the output of the Transformer model (decoding feature F c ), and finally add residual connection to make the convolution adjustment module have no side effects. The convolution adjustment module can be expressed as:

[0099]

[0100] In the application, the convolution adjustment module first multiplies the decoded features and edge features pixel by pixel to adjust the strength or weight of the decoded features to make them pay more attention to edge information. Then the adjusted features are added pixel by pixel to the edge features to integrate the edge feature information into the decoded features, thereby obtaining a richer feature representation. Finally, the original decoded features are combined with the adjusted features F m Residual connections are made to ensure the stability of the model and avoid information loss due to feature conditioning.

[0101] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0102] The present application also provides an image segmentation device for executing the steps in the above-mentioned image segmentation method embodiment. The image segmentation device can be a virtual appliance in a terminal device, which is run by a processor of the terminal device, or it can be the terminal device itself.

[0103] like Figure 5 As shown, the image segmentation device 100 provided in the embodiment of the present application includes:

[0104] The edge feature extraction module 101 is used to extract the edge of the input image to obtain the edge features of the input image;

[0105] The embedding vector generating module 102 is used to process the input image to obtain an image block embedding vector;

[0106] An image segmentation module 103 is used to segment the input image into n image block sequences of different resolutions and obtain a feature representation of each image block sequence;

[0107] The encoding module 104 is used to fuse the feature representation of the (i+1)th image block sequence with the output feature of the i-th encoder in a step-by-step encoding manner to obtain an encoding feature;

[0108] A decoding module 105, configured to decode the encoded feature to obtain a decoded feature;

[0109] An adjustment module 106, configured to fuse and adjust the decoding feature and the edge feature to obtain an adjustment feature;

[0110] A segmentation module 107, used for performing semantic segmentation on the adjustment features;

[0111] An output module 108, used to output the image obtained by the semantic segmentation;

[0112] Wherein, i=1, 2, ..., n-1, n is an integer greater than 1, and the feature representation of the first image block sequence is fused with the image block embedding vector.

[0113] In applications, each module in the image segmentation device may be a software program module, or may be implemented by different logic circuits integrated in a processor, or may be implemented by multiple distributed processors.

[0114] like Figure 6 As shown, the embodiment of the present application further provides a terminal device 200, including: at least one processor 201 ( Figure 6 Only one processor is shown in the figure), a memory 202, and a computer program 203 stored in the memory 202 and executable on at least one processor 201, wherein the processor 201 implements the steps in the above-mentioned various method embodiments when executing the computer program 203.

[0115] In the application, the terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 6 It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0116] In applications, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0117] In applications, the memory may be an internal storage unit of a terminal device in some embodiments, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as program codes of a computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0118] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0119] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0120] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0121] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0123] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0127] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image segmentation method, characterized in that: include: Performing edge extraction on an input image to obtain edge features of the input image; Processing the input image to obtain an image block embedding vector; Segment the input image into n image block sequences of different resolutions, and obtain a feature representation of each image block sequence; By means of step-by-step encoding, the feature representation of the (i+1)th image block sequence is fused with the output feature of the i-th encoder to obtain a coding feature; Decoding the encoded features to obtain decoded features; The decoding feature and the edge feature are fused and adjusted to obtain an adjustment feature; performing semantic segmentation on the adjustment features; Outputting the image obtained by the semantic segmentation; Wherein, i=1, 2, ..., n-1, n is an integer greater than 1, and the feature representation of the first image block sequence is fused with the image block embedding vector.

2. The image segmentation method according to claim 1, characterized in that: The step of extracting the edge of the input image to obtain the edge features of the input image includes: The input image is input into one or more convolutional layers for edge extraction to obtain edge features of the input image.

3. The image segmentation method according to claim 1, characterized in that: The step of processing the input image to obtain an image block embedding vector comprises: Divide the input image into multiple image blocks; Performing linear transformation processing on the multiple image blocks to obtain image block embedding vectors.

4. The image segmentation method according to claim 1, characterized in that: The step of dividing the input image into n image block sequences with different resolutions and obtaining a feature representation of each image block sequence includes: Dividing the input image into n image block sequences of different resolutions; Linear transformation and normalization processing are performed on the n image block sequences respectively to obtain feature representations of the n image block sequences.

5. The image segmentation method according to claim 1, characterized in that: The step of fusing the feature representation of the i+1th image block sequence with the output feature of the i-th encoder in a step-by-step encoding manner to obtain the encoding feature comprises: After the image block embedding vector is input into the first encoder for encoding, it is linked with the first image block sequence to obtain a first down-sampling link feature; Performing a linear transformation on the first down-sampled link feature to obtain an output feature of a first encoder; Downsample the output features of the i-th encoder; Inputting the result of the downsampling process to the i+1th encoder for encoding, and then linking it with the feature representation of the i+1th image block sequence input to obtain the i+1th downsampling link feature; Performing a linear transformation process on the i+1th down-sampled link feature to obtain an output feature of the i+1th encoder; After downsampling the output features of the nth encoder, the features are input into one or more sliding window self-attention modules for fusion to obtain encoding features.

6. The image segmentation method according to claim 1, characterized in that: The decoding of the encoded features to obtain decoded features includes: By decoding in a step-by-step manner, the output features of the ni-th encoder are fused with the output features of the ith decoder to obtain decoding features.

7. The image segmentation method according to claim 6, characterized in that: The step of fusing the output features of the ni-th encoder and the output features of the ith decoder by decoding step by step to obtain the decoding features includes: Performing a first upsampling process on the encoded feature, and performing a jump link with the output feature of the nth encoder to obtain a first upsampling link feature; Inputting the first up-sampled link feature into the first decoder for decoding to obtain an output feature of the first decoder; Performing a first upsampling process on the output features of the i-th decoder; Perform a jump link on the result of the first upsampling process and the output feature of the ni-th encoder to obtain the (i+1)-th upsampling link feature; Inputting the i+1th upsampled link feature into the i+1th decoder for decoding, to obtain an output feature of the i+1th decoder; A second upsampling process is performed on the output features of the nth decoder to obtain the decoded features.

8. The image segmentation method according to claim 1, characterized in that: The step of fusing and adjusting the decoding feature and the edge feature to obtain an adjustment feature includes: Using the formula fusing and adjusting the decoding feature and the edge feature to obtain an adjusted feature; in, represents pixel-by-pixel multiplication, represents pixel-by-pixel addition, F c is the decoding feature, F t is the edge feature, F m To adjust the characteristics.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image segmentation method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 8 are implemented.

Images