Image segmentation method and device, image segmentation equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HONG KONG POLYTECHNIC UNIVERSITY (SHENZHEN) FRONTIER TECHNOLOGY INNOVATION CENTER CO LTD
- Filing Date
- 2026-03-17
- Publication Date
- 2026-07-03
Smart Images
Figure CN122336271A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing, and more particularly to an image segmentation method, apparatus, device, and storage medium. Background Technology
[0002] Image segmentation is one of the core technologies in image processing. Its purpose is to separate regions with specific semantic meanings in an image, such as target objects and background regions. It is widely used in computer vision, medical image analysis, autonomous driving, intelligent monitoring, and many other fields. With the development of artificial intelligence technology, deep learning-based image segmentation models have become mainstream. Among them, large-scale vision models, with their powerful global feature extraction capabilities, have demonstrated excellent performance in image segmentation of complex scenes.
[0003] However, large-scale visual models are typically designed for general-purpose scenarios and struggle to adapt to complex applications in specific contexts. For instance, in skin cancer screening within medical image analysis, it's crucial to accurately segment lesion areas from microscopic skin images to aid in determining the boundaries and extent of malignant lesions such as melanoma. General-purpose large-scale visual models often fall short of the required segmentation accuracy. While fine-tuning the entire large-scale visual model can improve accuracy to some extent, the computational cost is extremely high. Furthermore, for applications like healthcare where sample resources are scarce, directly fine-tuning the entire large-scale visual model yields poor results; even after fine-tuning, the segmentation accuracy of the large-scale visual model still falls short of practical requirements. Summary of the Invention
[0004] The main objective of this application is to provide an image segmentation method, apparatus, device, and storage medium, aiming to solve the problem that the segmentation accuracy of image segmentation methods is poor and cannot meet practical needs.
[0005] To achieve the above objectives, a first aspect of this application proposes an image segmentation method applied to an electronic device. The electronic device integrates an image segmentation model, which includes a large visual model and a local optimization module. The large visual model includes N sequentially connected coding layers. The local optimization module includes a local feature cue encoder sequentially connected and N local optimization layers corresponding one-to-one with the N coding layers. Each local optimization layer is connected to the output of its corresponding coding layer, and N is a positive integer. The method includes: The image to be segmented is input into the image segmentation model, and features are extracted from the image to be segmented through the first coding layer to obtain the large model features output by the first coding layer; and features are extracted from the image to be segmented through the local feature cue encoder to obtain the cue sequence input by the first local tuning layer. Feature extraction is performed sequentially through the remaining encoding layers, and feature optimization and cue sequence updates are performed sequentially through the N local optimization layers. The input features of the i-th encoding layer are determined based on the large model features output by the (i-1)-th encoding layer and the optimized features output by the (i-1)-th local optimization layer. The optimized features output by the (i-1)-th local optimization layer are determined based on the output features of the (i-1)-th encoding layer and the cue sequence input by the (i-1)-th local optimization layer. i is an integer greater than 1 and less than N. Local cues are obtained based on the large model features output by the Nth coding layer and the cue sequence output by the Nth local tuning layer; wherein the cue sequence output by the Nth local tuning layer is determined based on the cue sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer. Based on the local prompts, a segmentation mask for the image to be segmented is generated, and the image to be segmented is segmented according to the segmentation mask.
[0006] In some implementations, the image segmentation model further includes a global guidance cue encoder and a mask decoder; The step of generating a segmentation mask for the image to be segmented based on the local prompts includes: Obtain the large model features output by the global coding layer among the N coding layers; The large model features output from the global encoding layer are input into the global guidance and prompt encoder to obtain global prompts; The local and global hints are input into the mask decoder to obtain the segmentation mask.
[0007] In some implementations, the number of global coding layers is M, and the global guidance and prompting encoder includes cross-attention layers that correspond one-to-one with the M global coding layers; M is a positive integer, and M is less than or equal to N; The step of inputting the large model features output from the global encoding layer into the global guidance and cue encoder to obtain global cue includes: For the large model features output by each global encoding layer, a cross-attention operation is performed on the large model features through the global cue sequence in the cross-attention layer corresponding to the global encoding layer to obtain the first cross feature; The first cross features corresponding to the features of each of the large models are superimposed, and key features are extracted from the superimposed features to obtain the global prompt.
[0008] In some implementations, the input features of the i-th coding layer are determined based on the large model features output by the (i-1)-th coding layer and the tuning features output by the (i-1)-th local tuning layer, including: The large model features output by the (i-1)th coding layer are added to the tuning features output by the (i-1)th local tuning layer to obtain the input features of the i-th coding layer.
[0009] In some implementations, the tuning features output by the (i-1)th local tuning layer are determined based on the output features of the (i-1)th coding layer and the cue sequence input to the (i-1)th local tuning layer, including: Based on the prompt sequence input by the (i-1)th local tuning layer, the large model features output by the (i-1)th coding layer are tuned through the (i-1)th local tuning layer to obtain the tuned features output by the (i-1)th local tuning layer.
[0010] In some implementations, obtaining local cues based on the large model features output by the Nth coding layer and the cue sequence output by the Nth local tuning layer includes: The large model features output by the Nth coding layer are added to the optimized features output by the Nth local optimization layer to obtain the added features; The summed features are convolved to obtain the first convolutional features; The cue sequence output by the Nth local tuning layer is convolved to obtain the second convolutional feature; The local cue is obtained by adding the first convolutional feature and the second convolutional feature.
[0011] In some implementations, the cue sequence output by the Nth local tuning layer is determined based on the cue sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer, including: A cross-attention operation is performed on the cue sequences output by the N-1 local tuning layers and the large model features output by the Nth coding layer to obtain the second cross features; Based on the second cross feature, the prompt sequence output by the (N-1)th local optimization layer is updated to obtain the prompt sequence output by the Nth local optimization layer.
[0012] To achieve the above objectives, a second aspect of this application proposes an image segmentation method apparatus applied to an electronic device. The electronic device integrates an image segmentation model, which includes a large visual model and a local optimization module. The large visual model includes N sequentially connected coding layers. The local optimization module includes a local feature cue encoder sequentially connected to the N coding layers and N local optimization layers corresponding one-to-one with the N coding layers. Each local optimization layer is connected to the output of its corresponding coding layer, and N is a positive integer. The device includes: The extraction module is used to input the image to be segmented into the image segmentation model, extract features from the image to be segmented through the first encoding layer to obtain the large model features output by the first encoding layer; and extract features from the image to be segmented through the local feature cue encoder to obtain the cue sequence input by the first local tuning layer. The optimization module is used to extract features sequentially through the remaining encoding layers, and to optimize features and update the cue sequence sequentially through the N local optimization layers. The input features of the i-th encoding layer are determined based on the large model features output by the (i-1)-th encoding layer and the optimized features output by the (i-1)-th local optimization layer. The optimized features output by the (i-1)-th local optimization layer are determined based on the output features of the (i-1)-th encoding layer and the cue sequence input by the (i-1)-th local optimization layer. i is an integer greater than 1 and less than N. The prompting module is used to obtain local prompts based on the large model features output by the Nth coding layer and the prompting sequence output by the Nth local tuning layer; wherein the prompting sequence output by the Nth local tuning layer is determined based on the prompting sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer. The segmentation module is used to generate a segmentation mask for the image to be segmented based on the local prompts, and to segment the image to be segmented based on the segmentation mask.
[0013] To achieve the above objectives, a third aspect of this application provides an image segmentation apparatus, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image segmentation method described in the first aspect.
[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image segmentation method described in the first aspect.
[0015] To achieve the above objectives, embodiments of this application may provide a computer program product, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to implement the image segmentation method described in the first aspect.
[0016] The image segmentation method, apparatus, device, and storage medium proposed in this application combine a large visual model with a lightweight local optimization module in parallel within the image segmentation model. By utilizing the inter-layer transfer and update mechanism of the cue sequence in the local optimization module, as well as the feedback of optimization features to the input of the coding layer, efficient fusion of global semantics and local details of the image to be segmented is achieved. This not only compensates for the shortcomings of the large visual model in edge capture, but also enhances the interaction and utilization of multi-scale features. It realizes low-cost and high-efficiency local adaptive enhancement of the large visual model, significantly improves the image segmentation accuracy of the image segmentation model, and meets the needs of practical applications. Specifically, in practical applications, the image to be segmented is input into the image segmentation model. Features are extracted layer by layer through the N coding layers of the large visual model within the image segmentation model to accurately capture the global semantic features of the portrait image. Simultaneously, through the one-to-one collaboration between the N local optimization layers of the local optimization module and the coding layers, and combined with the prompt sequence generated by the local feature cue encoder, the model is updated layer by layer, effectively compensating for the shortcomings of the large visual model in capturing local detail features. Finally, based on the large model features output from the Nth coding layer and the prompt sequence output from the Nth local optimization layer, local cueing is obtained. Based on this local cueing, a more accurate segmentation mask is generated. This segmentation mask is then used to segment the image to be segmented, significantly improving the segmentation accuracy of the image. Attached Figure Description
[0017] Figure 1 This is a schematic flowchart of the image segmentation method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of an image segmentation model provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the image segmentation method apparatus provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of the image segmentation device provided in the embodiments of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0021] Image segmentation is one of the core technologies in image processing. Its purpose is to separate regions with specific semantic meanings in an image, such as target objects and background regions. It is widely used in computer vision, medical image analysis, autonomous driving, intelligent monitoring, and many other fields. With the development of artificial intelligence technology, deep learning-based image segmentation models have become mainstream. Among them, large-scale vision models, with their powerful global feature extraction capabilities, have demonstrated excellent performance in image segmentation of complex scenes.
[0022] However, large-scale visual models are typically designed for general-purpose scenarios and struggle to adapt to complex applications in specific contexts. For instance, in skin cancer screening within medical image analysis, it's crucial to accurately segment lesion areas from microscopic skin images to aid in determining the boundaries and extent of malignant lesions such as melanoma. General-purpose large-scale visual models often fall short of the required segmentation accuracy. While fine-tuning the entire large-scale visual model can improve accuracy to some extent, the computational cost is extremely high. Furthermore, for applications like healthcare where sample resources are scarce, directly fine-tuning the entire large-scale visual model yields poor results; even after fine-tuning, the segmentation accuracy of the large-scale visual model still falls short of practical requirements.
[0023] Based on this, embodiments of this application provide an image segmentation method, apparatus, device, and storage medium, aiming to solve the problems of poor interpretability and low accuracy in image segmentation methods.
[0024] The image segmentation method, apparatus, device, and storage medium provided in this application are specifically described through the following embodiments. First, the image segmentation method in this application is described.
[0025] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0026] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0027] The image segmentation method, apparatus, device, and storage medium provided in this application relate to the field of image processing. The image segmentation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the image segmentation method, but is not limited to the above forms.
[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0029] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0030] Figure 1 This is a flowchart illustrating the image segmentation method provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 The image segmentation method provided in this application is applied to an electronic device. Specifically, the electronic device can be a server, personal computer, mobile terminal, or other device with data processing capabilities. The electronic device integrates an image segmentation model, which includes a large visual model and a local optimization module. The large visual model includes N sequentially connected encoding layers. The local optimization module includes a sequentially connected local feature cue encoder and N local optimization layers corresponding one-to-one with the N encoding layers. Each local optimization layer is connected to the output of its corresponding encoding layer, and N is a positive integer. The method may include, but is not limited to, steps S101 to S104.
[0031] Step S101: Input the image to be segmented into the image segmentation model, extract features from the image to be segmented through the first coding layer to obtain the large model features output by the first coding layer; and extract features from the image to be segmented through the local feature cue encoder to obtain the cue sequence input by the first local tuning layer.
[0032] In this step, the image to be segmented is input into the image segmentation model, and the initial processing flow of the large visual model and the local optimization module is started.
[0033] Specifically, on the one hand, the first encoding layer in the visual large model performs preliminary feature extraction on the image to be segmented, transforming the original pixel information of the image into a feature vector with certain semantic information, thus obtaining the large model features output by the first encoding layer; on the other hand, the local feature cue encoder in the local tuning module performs local feature extraction on the image to be segmented, focusing on capturing local details, boundary contours and other information in the image, and transforming these local features into a cue sequence in a specific format, which serves as the input to the first local tuning layer in the local tuning module.
[0034] It should be noted that the first coding layer is the first coding layer in a series of N coding layers, and the first local tuning layer is the first local tuning layer in a series of N local tuning layers.
[0035] Step S102: Feature extraction is performed sequentially through the remaining coding layers, and feature optimization and prompt sequence updates are performed sequentially through the N local optimization layers; wherein, the input feature of the i-th coding layer is determined based on the large model feature output by the (i-1)-th coding layer and the optimized feature output by the (i-1)-th local optimization layer; the optimized feature output by the (i-1)-th local optimization layer is determined based on the output feature of the (i-1)-th coding layer and the prompt sequence input by the (i-1)-th local optimization layer; i is an integer greater than 1 and i is less than N.
[0036] In this step, after the initial processing is completed, the collaborative iterative processing of the remaining coding layers of the large visual model and the N local tuning layers of the local tuning module is initiated. The specific process is as follows: The remaining coding layers in the large visual model (i.e., the 2nd to Nth coding layers) sequentially extract features from the input features. The input features of the i-th coding layer (where i is an integer greater than 1 and less than N) do not solely depend on the output of the previous coding layer, but are determined jointly by the large model features output from the (i-1)-th coding layer and the optimized features output from the (i-1)-th local optimization layer. This ensures that the feature extraction at each coding layer incorporates locally optimized detailed features, improving the accuracy of high-level semantic features.
[0037] Meanwhile, the N local optimization layers in the local optimization module sequentially perform feature optimization and cue sequence updates. The optimized features output by the (i-1)th local optimization layer can be obtained through cross-attention or feature fusion, based on the output features of the (i-1)th encoding layer (i.e., the large model features extracted by that encoding layer) and the cue sequence input to the (i-1)th local optimization layer (initially the cue sequence generated by the local feature cue encoder, and subsequently the cue sequence updated by the previous local optimization layer). By optimizing the features, local details are corrected for the large model features output by the corresponding encoding layer, compensating for the shortcomings of the large visual model in capturing local features.
[0038] After each local optimization layer completes the optimization process, the input prompt sequence is updated based on the current optimization result and the output features of the corresponding coding layer to obtain a new prompt sequence, which is then input into the next local optimization layer. This process achieves layer-by-layer optimization of the prompt sequence, enabling subsequent optimization processes to be based on more accurate local feature information.
[0039] Step S103: Obtain local prompts based on the large model features output by the Nth coding layer and the prompt sequence output by the Nth local tuning layer; wherein, the prompt sequence output by the Nth local tuning layer is determined based on the prompt sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer.
[0040] In this step, when the Nth encoding layer of the visual large model completes feature extraction and obtains the large model features output by the Nth encoding layer, the Nth local tuning layer of the local tuning module completes the cue sequence update and obtains the cue sequence output by the Nth local tuning layer.
[0041] Based on the large model features output from the Nth coding layer and the cue sequence output from the Nth local tuning layer, a local cue that can take into account both global semantics and local details is obtained through feature fusion processing. This local cue provides the core basis for the generation of the subsequent segmentation mask.
[0042] The cue sequence output by the Nth local tuning layer can be determined based on the updated cue sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer.
[0043] Step S104: Generate a segmentation mask for the image to be segmented based on the local prompts, and segment the image to be segmented based on the segmentation mask.
[0044] In this step, the generated local cues are input into the decoder or mask generation head to obtain a segmentation mask for the image to be segmented. The decoder can consist of multiple deconvolutional or upsampling layers, decoding the local cues into a segmentation mask with the same size as the original image.
[0045] Each pixel value in the segmentation mask represents the probability that the pixel belongs to a certain category or the binarized assignment result (0 or 1).
[0046] Electronic devices can process the image to be segmented based on the segmentation mask, such as extracting foreground objects from the background or coloring and labeling different semantic regions, thereby completing the final image segmentation task.
[0047] In this implementation, a large visual model is combined in parallel with a lightweight local optimization module in the image segmentation model. By utilizing the inter-layer transfer and update mechanism of the cue sequence in the local optimization module, as well as the feedback of the optimization features to the input of the coding layer, efficient fusion of global semantics and local details of the image to be segmented is achieved. This not only makes up for the shortcomings of the large visual model in edge capture, but also enhances the interaction and utilization of multi-scale features. It realizes low-cost and high-efficiency local adaptive enhancement of the large visual model, significantly improves the image segmentation accuracy of the image segmentation model, and meets the needs of practical applications.
[0048] In some implementations, the image segmentation model further includes a global guidance cue encoder and a mask decoder, and step S104 may also include the following steps S201 to S203: Step S201: Obtain the large model features output by the global coding layer in the N coding layers; Step S202: Input the large model features output by the global coding layer into the global guidance and prompt encoder to obtain global prompts; Step S203: Input the local hints and the global hints into the mask decoder to obtain the segmentation mask.
[0049] The number of global coding layers is M, and the global guidance and prompting encoder includes cross-attention layers that correspond one-to-one with the M global coding layers; M is a positive integer, and M is less than or equal to N; Step S202 may also include the following: For the large model features output by each global encoding layer, a cross-attention operation is performed on the large model features through the global cue sequence in the cross-attention layer corresponding to the global encoding layer to obtain the first cross feature; The first cross features corresponding to the features of each of the large models are superimposed, and key features are extracted from the superimposed features to obtain the global prompt.
[0050] In this implementation, the N coding layers of the visual large model include M global coding layers. For example, the visual large model has 10 coding layers, with the 2nd, 4th, 6th, and 8th being global coding layers.
[0051] Within the global guidance and prompt encoder, there is a pre-defined set of learnable global prompt sequences. These global prompt sequences can be viewed as a set of vectors containing semantic information about the query, aiming to extract the most useful information for the segmentation task from the large model features.
[0052] Furthermore, the large model features output by the global encoding layer are obtained, and the large model features output by the M global encoding layers are respectively input into the corresponding cross-attention layers in the global guidance and prompting encoder.
[0053] For each large model feature output by a global encoding layer, a cross-attention operation is performed on the large model feature using a pre-defined global cue sequence in the corresponding cross-attention layer. Key information related to the global cue sequence is extracted from the large model feature, while irrelevant information is filtered out, thus obtaining the first cross feature corresponding to that layer. After calculating the first cross features for all global encoding layers, the first cross features corresponding to each large model feature are superimposed to achieve the fusion of global features from different levels. Then, key features are extracted from the superimposed features to remove redundancy and refine information. This can be implemented using a multilayer perceptron (MLP), convolutional layer, or self-attention module. The most representative parts are then selected from the superimposed features to ultimately generate high-quality global cue.
[0054] Furthermore, both local and global cues are input into the mask decoder. The mask decoder uses its internal fusion mechanisms, such as cross-attention, feature concatenation, or gated fusion, to interact and fuse the local and global cues. The mask decoder uses the global cues to filter and correct local features, removing background noise, and predicts the foreground and background probabilities of the image to be segmented pixel-by-pixel based on the fused features, ultimately generating an accurate segmentation mask. The segmentation mask is used to mark the boundaries and extents of different semantic regions in the image, clearly distinguishing between target and background regions, and regions between different targets.
[0055] In this implementation, the segmentation mask integrates global macro guidance and local fine detail prompts, thus accurately segmenting the target object. That is, while ensuring the integrity of the object as a whole based on global prompts, it also ensures the clarity of the edges based on local prompts.
[0056] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the image segmentation model provided in an embodiment of this application. Icons such as flames (representing optimization) and snowflakes (representing freezing) emphasize the dynamic and frozen states of the modules. This architecture emphasizes the fusion of local feature enhancement and global cue guidance to achieve efficient image segmentation adaptation.
[0057] Specifically Figure 2 The upper middle section shows the process of the input image being processed by multiple region calculation modules (encoding layer 1, encoding layer 2... encoding layer N) and convolution modules to generate image features. Finally, the image features are input into the prompt decoder and processed by the mask decoder to generate the target segmentation mask (i.e., the processing of the large visual model).
[0058] The lower part describes a schematic diagram of dual-visual cue optimization. The first part is a schematic diagram of local feature cueing. The input image first passes through a local feature cue encoder to obtain local features as cue 0. Then, in each local optimization layer, the large model feature sequence (large model feature 1 to large model feature N) is optimized. The returned optimized features (optimized features 1 to optimized features N) are added element-wise with the large model features to generate a new cue sequence (cue 1 to cue N-1). Finally, the local optimized features are obtained through convolution and added element-wise with the image features.
[0059] The second part is the global guidance and prompt encoder, which takes the global features of the large model (i.e., the large model features output by the global encoding layer out of N encoding layers) as input and generates global prompts. This is combined with the local feature prompt optimization module to achieve dual visual prompt optimization. Finally, through element-wise addition operations and a mask decoder, a target segmentation mask is generated, supporting noise reduction and boundary refinement.
[0060] The local feature encoder can be a ResNet-34 encoder.
[0061] Each local optimization layer can be composed of a cross-attention layer and a small feedforward network connected in series. The large model features output from the visual large model encoding layer are cross-attentioned with the local cue sequence. The cross-attention features are then input into the feedforward network to obtain optimized features. The cue sequence is then input into a multi-layer perceptron layer for updating, preparing for subsequent feature optimization in the encoding layer.
[0062] The global guidance and prompt encoder includes M lightweight learnable parameters as global prompt sequences. The global encoder includes M parallel cross-attention layers and a small feedforward network. The M cross-attention layers respectively process the global features (i.e., large model features) output from the M global feature encoding layers. The global features are cross-attentioned with the corresponding global prompt sequences. The output features are superimposed and input into the feedforward network for key feature extraction and feature shape modification to obtain the final global prompt.
[0063] The mask decoder can employ residual join operations to element-wise add the global cue to the output of each decoding layer in the decoder.
[0064] It should be noted that when training the image segmentation model, the parameters of the large visual model can be frozen, and only the parameters of the lightweight local feature cueing tuning module and the global guidance cueing encoder can be trained. Therefore, it can also show good segmentation performance when facing application scenarios with small samples, quickly adapt to the specific segmentation tasks of the application scenario, and meet the segmentation accuracy requirements of the application scenario.
[0065] In some implementations, the input features of the i-th coding layer in step S102 are determined based on the large model features output by the (i-1)-th coding layer and the tuning features output by the (i-1)-th local tuning layer, and may include the following: The large model features output by the (i-1)th coding layer are added to the tuning features output by the (i-1)th local tuning layer to obtain the input features of the i-th coding layer.
[0066] In this implementation, the i-th The large model features output by a single encoding layer are features processed by mainstream visual large model networks and can contain global structural and macroscopic semantic information of the image to be segmented.
[0067] i-th The optimized features output by a local optimization layer are the features after processing by the local optimization layer. They are obtained by the local optimization layer using cue sequences to enhance the details of the features. The optimized features can include local high-frequency detail information such as edges and textures to supplement the missing details in the large model features.
[0068] In this implementation, the input features of the i-th encoding layer of the visual large model are obtained by adding the large model features output by the previous (i-1)-th encoding layer and the optimized features output by the corresponding (i-1)-th local optimization layer. This ensures that the locally optimized detailed features can effectively play a role in the feature extraction of subsequent encoding layers, thereby improving the accuracy of high-level semantic features.
[0069] In some implementations, the tuning features output by the (i-1)th local tuning layer in step S102 are determined based on the output features of the (i-1)th coding layer and the cue sequence input to the (i-1)th local tuning layer, and may include the following: Based on the prompt sequence input by the (i-1)th local tuning layer, the large model features output by the (i-1)th coding layer are tuned through the (i-1)th local tuning layer to obtain the tuned features output by the (i-1)th local tuning layer.
[0070] In this implementation, the i-th A cue sequence input to a local tuning layer can contain prior parameters that guide the visual model to focus on specific local regions, enhance edge responses, or restore texture details.
[0071] The tuning features output by the (i-1)th local tuning layer are determined based on the output features of the (i-1)th coding layer and its input cue sequence. Specifically, they can be obtained by performing a cross-attention operation on the large model features output by the coding layer through the input cue sequence.
[0072] In this implementation, cue sequences are used to fine-tune the features of the large visual model. This allows the large visual model to actively intervene in and optimize the feature extraction process instead of passively accepting fixed features extracted by convolutional or attention layers. This not only solves the problem of local detail loss in deep networks but also enhances the visual model's adaptability to local differences in different images through a dynamic cue mechanism, thereby significantly improving the accuracy of the final segmentation results.
[0073] In some implementations, step S103 may also include the following: The large model features output by the Nth coding layer are added to the optimized features output by the Nth local optimization layer to obtain the added features; The summed features are convolved to obtain the first convolutional features; The cue sequence output by the Nth local tuning layer is convolved to obtain the second convolutional feature; The local cue is obtained by adding the first convolutional feature and the second convolutional feature.
[0074] In this implementation, the large model features output by the Nth coding layer and the optimized features output by the Nth local optimization layer are added element-wise to obtain the added features.
[0075] Furthermore, in order to convert the additive features into a format suitable for use as prompt information, the additive features are convolved to obtain the first convolutional features. Specifically, a 1*1 convolution kernel or a convolution kernel of other sizes can be used for convolution to adjust the number of channels of the additive features so that they meet the input dimension requirements of subsequent processing or decoders.
[0076] Similarly, the cue sequence output by the Nth local optimization layer is convolved to obtain the second convolutional feature.
[0077] Furthermore, the first and second convolutional features are added element-wise to obtain local cues. This addition is also for optimizing the first convolutional feature output by the convolutional layers in the large visual model.
[0078] In this implementation, a dual feature injection mechanism is constructed by adding the large model features output by the encoder at layer N to the optimized features output by the local optimization layer, and then adding them again to the processed cue sequence after convolution. This effectively integrates the deep semantics of the large visual model, the detail gain of the local optimization module, and the guiding information of the cue sequence, resulting in generated local cues with higher information purity and representational ability, thereby significantly improving the accuracy of the image segmentation model in capturing image details.
[0079] In some implementations, the cue sequence output by the Nth local tuning layer in step S103 is determined based on the cue sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer, and may include the following: A cross-attention operation is performed on the cue sequences output by the N-1 local tuning layers and the large model features output by the Nth coding layer to obtain the second cross features; Based on the second cross feature, the prompt sequence output by the (N-1)th local optimization layer is updated to obtain the prompt sequence output by the Nth local optimization layer.
[0080] In this implementation, the prompt sequence output by the (N-1)th local optimization layer can be used as the query, and the large model features output by the Nth encoding layer can be used as the key and value. Specifically, the local prior query in the prompt sequence output by the (N-1)th local optimization layer is used to query the relevant semantic information in the large model features output by the Nth encoding layer, and the attention output obtained after calculation is the second cross feature.
[0081] For example, if the prompt sequence output by the (N-1)th local tuning layer contains the prompt "melanoma edge", the cross-attention mechanism will retrieve global features that are strongly correlated with the semantic category "melanoma edge" from the large model features output by the Nth encoding layer.
[0082] Furthermore, after obtaining the second cross feature, the prompt sequence output by the (N-1)th local optimization layer can be updated using a multilayer perceptron to obtain the prompt sequence output by the Nth local optimization layer.
[0083] In this implementation, since the feature scales extracted by different coding layers are different, by combining the large model features of the current layer for cross-attention operation, the cue sequence can dynamically adjust its focus and always match the feature expression of the current layer. In the process of updating the cue sequence, historical information (N-1 layer) is retained while new information (N layer) is fused, so that the model can be like a memory, continuously enriching the description of local details as the processing depth increases, ensuring that the final generated local cue contains the most complete and accurate prior details.
[0084] Please see Figure 3This application also provides an image segmentation method apparatus 600, applied to an electronic device. The electronic device integrates an image segmentation model, which includes a large visual model and a local optimization module. The large visual model includes N sequentially connected coding layers. The local optimization module includes a local feature cue encoder sequentially connected, and N local optimization layers corresponding one-to-one with the N coding layers. Each local optimization layer is connected to the output of its corresponding coding layer. N is a positive integer. This allows the implementation of the above-mentioned image segmentation method. The image segmentation method apparatus 600 includes: The extraction module 601 is used to input the image to be segmented into the image segmentation model, extract features from the image to be segmented through the first coding layer to obtain the large model features output by the first coding layer; and extract features from the image to be segmented through the local feature cue encoder to obtain the cue sequence input by the first local tuning layer. The optimization module 602 is used to extract features sequentially through the remaining coding layers, and to optimize features and update the cue sequence sequentially through the N local optimization layers; wherein, the input features of the i-th coding layer are determined based on the large model features output by the (i-1)-th coding layer and the optimized features output by the (i-1)-th local optimization layer; the optimized features output by the (i-1)-th local optimization layer are determined based on the output features of the (i-1)-th coding layer and the cue sequence input by the (i-1)-th local optimization layer; i is an integer greater than 1 and i is less than N; The prompting module 603 is used to obtain local prompts based on the large model features output by the Nth coding layer and the prompting sequence output by the Nth local tuning layer; wherein the prompting sequence output by the Nth local tuning layer is determined based on the prompting sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer. The segmentation module 604 is used to generate a segmentation mask for the image to be segmented based on the local prompts, and to segment the image to be segmented based on the segmentation mask.
[0085] The specific implementation of the image segmentation method device 600 is basically the same as the specific embodiment of the image segmentation method described above, and will not be repeated here.
[0086] This application also provides an image segmentation device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image segmentation method described above. This image segmentation device can be any smart terminal, including desktop computers, tablets, mobile phones, and in-vehicle computers.
[0087] Please see Figure 4 , Figure 4This is a schematic diagram of the hardware structure of the image segmentation device provided in an embodiment of this application. The image segmentation device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the image segmentation method of the embodiments of this application. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.
[0088] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image segmentation method.
[0089] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0090] Alternatively, this application embodiment can provide a computer program product for implementation, wherein the instructions in the computer program product, when executed by the processor of an electronic device, cause the electronic device to implement the image segmentation method in the above embodiment.
[0091] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0092] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0094] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0095] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0096] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0097] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0098] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0099] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0101] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. An image segmentation method, characterized in that, The method is applied to electronic devices, which integrate an image segmentation model. The image segmentation model includes a large visual model and a local optimization module. The large visual model includes N coding layers connected in series. The local optimization module includes a local feature cue encoder connected in series and N local optimization layers corresponding one-to-one with the N coding layers. Each local optimization layer is connected to the output of the corresponding coding layer, and N is a positive integer. The method includes: The image to be segmented is input into the image segmentation model, and features are extracted from the image to be segmented through the first coding layer to obtain the large model features output by the first coding layer; and features are extracted from the image to be segmented through the local feature cue encoder to obtain the cue sequence input by the first local tuning layer. Feature extraction is performed sequentially through the remaining encoding layers, and feature optimization and cue sequence updates are performed sequentially through the N local optimization layers. The input features of the i-th encoding layer are determined based on the large model features output by the (i-1)-th encoding layer and the optimized features output by the (i-1)-th local optimization layer. The optimized features output by the (i-1)-th local optimization layer are determined based on the output features of the (i-1)-th encoding layer and the cue sequence input by the (i-1)-th local optimization layer. i is an integer greater than 1 and less than N. Local cues are obtained based on the large model features output by the Nth coding layer and the cue sequence output by the Nth local tuning layer; wherein the cue sequence output by the Nth local tuning layer is determined based on the cue sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer. Based on the local prompts, a segmentation mask for the image to be segmented is generated, and the image to be segmented is segmented according to the segmentation mask.
2. The method according to claim 1, characterized in that, The image segmentation model also includes a global guidance and cueing encoder and a mask decoder; The step of generating a segmentation mask for the image to be segmented based on the local prompts includes: Obtain the large model features output by the global coding layer among the N coding layers; The large model features output from the global encoding layer are input into the global guidance and prompt encoder to obtain global prompts; The local and global hints are input into the mask decoder to obtain the segmentation mask.
3. The method according to claim 2, characterized in that, The number of global coding layers is M, and the global guidance and prompting encoder includes cross-attention layers that correspond one-to-one with the M global coding layers; M is a positive integer, and M is less than or equal to N; The step of inputting the large model features output from the global encoding layer into the global guidance and cue encoder to obtain global cue includes: For the large model features output by each global encoding layer, a cross-attention operation is performed on the large model features through the global cue sequence in the cross-attention layer corresponding to the global encoding layer to obtain the first cross feature; The first cross features corresponding to the features of each of the large models are superimposed, and key features are extracted from the superimposed features to obtain the global prompt.
4. The method according to claim 1, characterized in that, The input features of the i-th coding layer are determined based on the large model features output by the (i-1)-th coding layer and the tuning features output by the (i-1)-th local tuning layer, including: The large model features output by the (i-1)th coding layer are added to the tuning features output by the (i-1)th local tuning layer to obtain the input features of the i-th coding layer.
5. The method according to claim 1, characterized in that, The tuning features output by the (i-1)th local tuning layer are determined based on the output features of the (i-1)th coding layer and the cue sequence input to the (i-1)th local tuning layer, including: Based on the prompt sequence input by the (i-1)th local tuning layer, the large model features output by the (i-1)th coding layer are tuned through the (i-1)th local tuning layer to obtain the tuned features output by the (i-1)th local tuning layer.
6. The method according to claim 1, characterized in that, The step of obtaining local cues based on the large model features output by the Nth coding layer and the cue sequence output by the Nth local tuning layer includes: The large model features output by the Nth coding layer are added to the optimized features output by the Nth local optimization layer to obtain the added features; The summed features are convolved to obtain the first convolutional features; The cue sequence output by the Nth local tuning layer is convolved to obtain the second convolutional feature; The local cue is obtained by adding the first convolutional feature and the second convolutional feature.
7. The method according to claim 1, characterized in that, The cue sequence output by the Nth local tuning layer is determined based on the cue sequence output by the (N-1)th local tuning layer and the large model features output by the Nth coding layer, including: A cross-attention operation is performed on the cue sequences output by the N-1 local tuning layers and the large model features output by the Nth coding layer to obtain the second cross features; Based on the second cross feature, the prompt sequence output by the (N-1)th local optimization layer is updated to obtain the prompt sequence output by the Nth local optimization layer.
8. An image segmentation apparatus, characterized in that, The method is applied to electronic devices, which integrate an image segmentation model. The image segmentation model includes a large visual model and a local optimization module. The large visual model includes N coding layers connected in series. The local optimization module includes a local feature cue encoder connected in series and N local optimization layers corresponding one-to-one with the N coding layers. Each local optimization layer is connected to the output of the corresponding coding layer, and N is a positive integer. The device includes: The extraction module is used to input the image to be segmented into the image segmentation model, extract features from the image to be segmented through the first encoding layer to obtain the large model features output by the first encoding layer; and extract features from the image to be segmented through the local feature cue encoder to obtain the cue sequence input by the first local tuning layer. The optimization module is used to extract features sequentially through the remaining encoding layers, and to optimize features and update the cue sequence sequentially through the N local optimization layers. The input features of the i-th encoding layer are determined based on the large model features output by the (i-1)-th encoding layer and the optimized features output by the (i-1)-th local optimization layer. The optimized features output by the (i-1)-th local optimization layer are determined based on the output features of the (i-1)-th encoding layer and the cue sequence input by the (i-1)-th local optimization layer. i is an integer greater than 1 and less than N. The prompting module is used to obtain local prompts based on the large model features output by the Nth encoding layer and the prompting sequence output by the Nth local tuning layer; wherein the prompting sequence output by the Nth local tuning layer is determined based on the prompting sequence output by the (N-1)th local tuning layer and the large model features output by the Nth encoding layer. The segmentation module is used to generate a segmentation mask for the image to be segmented based on the local prompts, and to segment the image to be segmented based on the segmentation mask.
9. An image segmentation device, characterized in that, The image segmentation device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image segmentation method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image segmentation method according to any one of claims 1 to 7.