Image segmentation method and device, equipment and storage medium

Abstract

The invention provides an image segmentation method and device, equipment and a storage medium, and the method comprises the steps: obtaining a to-be-segmented image and an operation prompt of a user on the to-be-segmented image; inputting a to-be-segmented image and an operation prompt into an encoder group, and respectively generating an image feature and a prompt feature through the encoder group; acquiring image features and prompt features through a classification module, extracting vocabulary information in the prompt features, outputting the probability that the image features belong to the category of the vocabulary information by using the classification module, and calling corresponding knowledge graph information as auxiliary features when the probability is lower than a threshold value; and obtaining the image features, the prompt features and the auxiliary features through a segmentation interpreter, combining the features into fusion features, and outputting a segmentation mask through the fusion features to obtain a segmentation result. The vocabulary information is used as a text prompt to eliminate a large amount of redundancy of the original long sentences, the classification module is introduced to pre-evaluate whether the vocabulary information can divide objects in the image, and the image segmentation generalization effect is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, device, equipment and storage medium. Background Art

[0002] Image segmentation is a basic task in the field of image processing, which aims to divide an image into multiple regions or objects with clear meanings. The goal is to extract important information from the image to facilitate subsequent more advanced analysis and processing, such as object recognition, tracking, classification, etc.

[0003] Among the related technologies, traditional image segmentation methods include classical algorithms and mathematical models, such as global threshold method, local threshold method, Canny edge detection method, clustering method, etc. These technologies rely on low-level features of the image and some prior knowledge for segmentation, and the segmentation accuracy is far lower than that based on deep learning methods.

[0004] Among deep learning models, SAM (Segment Anything Model) is a pioneering segmentation model developed by Meta AI. Different from traditional segmentation models, SAM supports multiple prompting methods such as points, boxes, text, etc., allowing users to flexibly control the segmentation results. However, the above technical means have the following obvious defects: On the one hand, the weak generalization ability of the SAM model itself is an important limitation, especially when dealing with objects that have not participated in training or complex scenes, it is difficult to effectively identify and segment; on the other hand, text prompts in the form of long sentences depend on the quality of user input. After using the natural semantic processing model to process long sentences into short sentences, if the long sentences contain meaningless and redundant limiting modifications, spending extra energy to process them cannot effectively improve the accuracy of segmentation.

[0005] Based on the above analysis of the development status of this technical field, the existing technology lacks a solution for using vocabulary as prompt text and defining a classification module to improve the generalization ability of the SAM model. Summary of the invention

[0006] The purpose of the present invention is to provide an image segmentation method, device, equipment and storage medium, aiming to solve the above-mentioned problems in the prior art.

[0007] According to a first aspect of an embodiment of the present invention, there is provided an image segmentation method, which is implemented based on a SAM segmentation model and includes:

[0008] Obtain the image to be segmented and the user's operation prompts on the image to be segmented, including:

[0009] Obtain an image to be segmented after resolution preprocessing;

[0010] The acquisition includes click information of clicking on the region of interest of the image, selection box information of marking the target with a matrix box, and vocabulary information used to describe the target as operation prompts.

[0011] The image to be segmented and the operation prompt are input into the encoder group, and the image features and prompt features are generated by the encoder group respectively, including:

[0012] Generate an embedded representation of the image to be segmented through an image encoder, perform a convolution operation on the embedded representation, and generate corresponding image features;

[0013] A two-dimensional prompt map of click information, a binary mask map of selection box information, and a text embedding of vocabulary information are generated through a prompt encoder, and the two-dimensional prompt map, binary mask map and text embedding are used together as prompt features.

[0014] The image features and prompt features are obtained through the classification module, the vocabulary information in the prompt features is extracted, and the classification module is used to output the probability that the image features belong to the vocabulary information category. When the probability is lower than the threshold, the corresponding knowledge graph information is called as an auxiliary feature, which includes:

[0015] The image features are input into the output layer of the classification module, wherein the classification module is a machine learning model that outputs the probability distribution of each category, indexes the vocabulary information in the probability distribution, and obtains the probability corresponding to the vocabulary information category.

[0016] When the probability is lower than a threshold, a pre-established knowledge graph including vocabulary information is obtained, neighbor nodes with the vocabulary information as the center are obtained from the knowledge graph, and a preset number of neighbor nodes are randomly selected as auxiliary features.

[0017] The image features, prompt features and auxiliary features are obtained through the segmentation interpreter and combined into fusion features. The segmentation mask is output through the fusion features to obtain the segmentation result, which includes:

[0018] Combine the prompt features and auxiliary features into the total features;

[0019] The cross-attention mechanism is used to perform bidirectional interaction between image features and total features to obtain attention image features and attention total features, which are then concatenated into fusion features.

[0020] Based on the fusion features, the segmentation mask of each pixel in the image to be segmented is output, and the segmentation result is output according to the category of the segmentation mask.

[0021] According to a second aspect of an embodiment of the present invention, an image segmentation device is provided, and the method is implemented based on a SAM segmentation model, comprising:

[0022] The acquisition module is used to obtain the image to be segmented and the user's operation prompts on the image to be segmented, specifically for:

[0023] Obtain an image to be segmented after resolution preprocessing;

[0024] The acquisition includes click information of clicking on the region of interest of the image, selection box information of marking the target with a matrix box, and vocabulary information used to describe the target as operation prompts.

[0025] The encoder module is used to input the image to be segmented and the operation prompt into the encoder group, and generate image features and prompt features through the encoder group, specifically for:

[0026] Generate an embedded representation of the image to be segmented through an image encoder, perform a convolution operation on the embedded representation, and generate corresponding image features;

[0027] A two-dimensional prompt map of click information, a binary mask map of selection box information, and a text embedding of vocabulary information are generated through a prompt encoder, and the two-dimensional prompt map, binary mask map and text embedding are used together as prompt features.

[0028] A classification module is used to obtain image features and prompt features through the classification module, extract vocabulary information in the prompt features, and use the classification module to output the probability that the image features belong to the vocabulary information category. When the probability is lower than a threshold, the corresponding knowledge graph information is called as an auxiliary feature;

[0029] The image features are input into the output layer of the classification module, wherein the classification module is a machine learning model that outputs the probability distribution of each category, indexes the vocabulary information in the probability distribution, and obtains the probability corresponding to the vocabulary information category.

[0030] When the probability is lower than a threshold, a pre-established knowledge graph including vocabulary information is obtained, neighbor nodes with the vocabulary information as the center are obtained from the knowledge graph, and a preset number of neighbor nodes are randomly selected as auxiliary features.

[0031] The decoder module is used to obtain image features, prompt features and auxiliary features through the segmentation interpreter, and combine them into fusion features. The segmentation mask is output through the fusion features to obtain the segmentation result. It is specifically used for:

[0032] Combine the prompt features and auxiliary features into the total features;

[0033] The cross-attention mechanism is used to perform bidirectional interaction between image features and total features to obtain attention image features and attention total features, which are then concatenated into fusion features.

[0034] Based on the fusion features, the segmentation mask of each pixel in the image to be segmented is output, and the segmentation result is output according to the category of the segmentation mask.

[0035] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the image segmentation method provided in the first aspect of the present disclosure are implemented.

[0036] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a program for implementing information transmission is stored, and when the program is executed by a processor, the steps of the image segmentation method provided in the first aspect of the present disclosure are implemented.

[0037] The technical solution provided by the embodiment of the present invention includes the following beneficial effects: using vocabulary information as text prompts to eliminate a large amount of redundancy contained in the original long sentence and reduce the computational consumption of using the natural language processing model; on the basis of the original SAM segmentation model, introducing a custom classification module for pre-evaluation of whether objects can be divided in the image based on vocabulary information. If not, more accurate knowledge graph information is used as auxiliary features, which greatly improves the generalization effect of image segmentation.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 is a flow chart of an image segmentation method according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of the SAM model architecture of an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of an image segmentation device according to an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0045] Method Embodiment

[0046] According to an embodiment of the present invention, an image segmentation method is provided. Figure 1 is a flow chart of an image segmentation method according to an embodiment of the present invention. The method is implemented based on the SAM segmentation model. The embodiment of the present invention adds a classification module based on the original SAM segmentation model architecture, such as Figure 1 As shown, the image segmentation method according to an embodiment of the present invention specifically includes:

[0047] In step S110, the image to be segmented and the user's operation prompt on the image to be segmented are obtained, which specifically includes:

[0048] Obtaining the image to be segmented after resolution preprocessing, where resolution preprocessing can improve the clarity of the image;

[0049] The acquisition includes click information of clicking on the region of interest of the image, selection box information of marking the target with a matrix box, and vocabulary information used to describe the target as an operation prompt. In actual application, the user can select one or more types of operation prompts according to actual conditions. The embodiment of the present invention focuses on optimizing the vocabulary information part, so the application scenario of the present invention is the scheme using the above three types of operation prompts;

[0050] Click information refers to the user clicking one or more points in the image to be segmented to indicate the object to be segmented, selection box information refers to the user drawing a rectangular box including the target object in the image to be segmented, and vocabulary information refers to the text describing the target object;

[0051] In an embodiment of the present invention, vocabulary information is used to replace common text information in a field;

[0052] The original long sentence itself is also intended to improve the generalization ability of the model, but the long sentences corresponding to the text information often contain a lot of redundancy and require additional word segmentation operations that increase the amount of computation. The limitations of long sentences are often modifications. Inaccurate limitations may increase the difficulty of segmentation model recognition. For example, for scissors of brand A, "brand A" has little effect on improving segmentation accuracy. Inaccurate descriptions not only fail to improve generalization ability, but also bring additional computational losses.

[0053] It should be noted that in the field of image segmentation, vocabulary information refers to nouns, such as "cat", "table", "scissors", etc.

[0054] In step S120, the image to be segmented and the operation prompt are input into the encoder group, and the encoder group generates image features and prompt features respectively, specifically including:

[0055] Generate an embedded representation of the image to be segmented through an image encoder, perform a convolution operation on the embedded representation, and generate corresponding image features;

[0056] In the embodiment of the present invention, the image encoder extracts multi-scale features of the image to be segmented based on the powerful ViT architecture, so that the embedded representation can capture rich information in the image. The multi-scale features have different spatial resolutions and semantic levels, and can represent a variety of information from local details to global structures.

[0057] Generate a two-dimensional prompt map of click information, generate a binary mask map of selection box information, generate text embedding of vocabulary information through a prompt encoder, and use the two-dimensional prompt map, binary mask map and text embedding together as prompt features;

[0058] For click information, the hint encoder generates a two-dimensional hint map based on the position of the click point, which usually has the same spatial resolution as the image feature. The position of the click point will be marked as a high activation value, while other positions will maintain a lower activation value. For box selection information, the hint encoder's binary mask map marks the pixels in the box area as 1 and the pixels outside the box as 0.

[0059] For lexical information, the prompt encoder aligns the text embedding vector with the image features. Specifically, the text embedding is expanded to the same spatial resolution as the image features through a broadcast operation to ensure that the lexical information can affect every position in the image.

[0060] In step S130, the image features and the prompt features are obtained through the classification module, the vocabulary information in the prompt features is extracted, and the probability that the image features belong to the vocabulary information category is output by the classification module. When the probability is lower than the threshold, the corresponding knowledge graph information is called as an auxiliary feature, which specifically includes:

[0061] The SAM model used for image segmentation has weak generalization ability, especially when dealing with objects or complex scenes that have not been seen during training. There are many kinds of objects in the world, and many objects may never appear in the training set. The existing improvement direction is often to modify the diversity of data in the training set, but this method has low controllability;

[0062] In the embodiment of the present invention, a classification module is customized to improve the generalization ability of the SAM model. That is, if the enhanced features include vocabulary information, taking the vocabulary information "scissors" as an example, a simple classification module is used to pre-identify whether the image features may contain the object "scissors". If the image features are likely to include "scissors", direct segmentation is sufficient, otherwise some information supplementary auxiliary recognition is required;

[0063] That is to say, a pre-assessment was conducted using the classification module;

[0064] The image features are input into the output layer of the classification module, wherein the classification module is a machine learning model. In the embodiment of the present invention, the specific machine learning model is not limited, and the model structure is simple and easy to use. However, since the encoder of SAM has extracted the image features, it is not necessary to input them into the complete machine learning model for image classification.

[0065] For example, convolutional neural networks also include feature extraction. It is only necessary to input image features into the output layer of the trained convolutional neural network to greatly improve the efficiency of image classification. It is necessary to consider converting image features into the form required by the corresponding output layer;

[0066] The probability distribution of each category is output by the machine learning model, and the softmax function is applied to the output layer to convert the classification score into a probability distribution;

[0067] Index the vocabulary information in the probability distribution to obtain the corresponding probability of the vocabulary information category. For example, the probability of the vocabulary information "scissors" is 0.8, the probability of "table" is 0.2, etc. Search "scissors" from a series of probability distributions and obtain the corresponding probability of 0.8;

[0068] When the probability is lower than the threshold, in the embodiment of the present invention, the threshold is 0.2, because for the image segmentation task, the object to be segmented is not necessarily the main body in the image features, and may also be located at the edge or in the occluded area. As long as it can be searched in the probability distribution, its lower probability value has little effect;

[0069] Since the probability of "scissors" in the embodiment of the present invention is 0.8, no auxiliary features are needed, which proves that the correct scissors shape can be divided in the image features. If the probability of "scissors" is 0.1 or cannot be found in the probability distribution, it proves that the current vocabulary information includes less information and needs auxiliary features.

[0070] Obtain a pre-established knowledge graph that includes vocabulary information, and obtain neighbor nodes with the vocabulary information as the center from the knowledge graph. Different from the restrictive modification of long sentences, the information in the knowledge graph is a more accurate description of the characteristics of the object itself, such as "scissors" have "handles" and "blades"; what characteristics do the ears, nose, eyes and other parts of a "cat" have, etc.;

[0071] Using objective and detailed descriptions can effectively assist the model in understanding the shape of objects, infer the location of the objects to be segmented, and guide the segmentation process.

[0072] Randomly select a preset number of neighbor nodes as auxiliary features.

[0073] In step S140, the image features, the prompt features and the auxiliary features are obtained by the segmentation interpreter and combined into fusion features. The segmentation mask is output by the fusion features to obtain the segmentation result, which specifically includes:

[0074] The prompt feature and the auxiliary feature are combined into the total feature, and the prompt feature is used as the total feature when the auxiliary feature is not needed;

[0075] Use the cross-attention mechanism to perform bidirectional interaction between image features and total features, obtain the attention image features and the attention total features, and splice the attention image features and the attention total features into fusion features; in order to enable effective interaction between image features and total features, align the two to different dimensions, which can be achieved through linear transformation;

[0076] Based on the fusion features, the segmentation mask of each pixel in the image to be segmented is output. The segmentation mask is also in binary form, which is used to indicate whether each pixel belongs to the target object. The shape and scale of the segmentation mask are the same as the input image to be segmented. The segmentation result is output according to the category of the segmentation mask.

[0077] In the embodiment of the present invention, based on the original image encoder, prompt encoder and segmentation interpreter of SAM, a classification module is added to serve as an improved SAM model.

[0078] The above technical solutions of the embodiments of the present invention are illustrated with reference to the following drawings.

[0079] Figure 2 Schematic diagram of the SAM model architecture of an embodiment of the present invention, such as Figure 2As shown, an improved SAM model architecture is demonstrated, which obtains the embedded representation of the image to be segmented output by the image encoder, and obtains image features through conv convolution; obtains click information, vocabulary information and text embedding as prompt features through the prompt encoder; the original SAM model directly sends the image features and prompt features to the segmentation interpreter, while the embodiment of the present invention customizes a classification module for pre-evaluation of whether the object can be divided in the image based on the vocabulary information. If not, the knowledge graph information is used as an auxiliary feature; and finally the segmentation result is obtained through the segmentation interpreter.

[0080] To sum up, in response to the existing problems, this invention proposes an image segmentation method, which is based on the SAM model; vocabulary information is used as text prompts to eliminate a large amount of redundancy and modification contained in the original long sentence, and reduce the computational consumption of long sentence segmentation using the natural language processing model; on the basis of the original SAM segmentation model, a custom classification module is introduced to pre-evaluate whether the object can be divided in the image based on the vocabulary information. If not, the knowledge graph information is used as an auxiliary feature to assist in inferring the position of the object in the image, so as to avoid the single vocabulary information being not comprehensive enough. The knowledge graph information is more accurate than the modification in the original long sentence, which can effectively alleviate the problem of weak generalization ability of the model; since the encoder has obtained the image features, only the output layer is used for prediction inside the classification module to minimize the impact of the classification module on the overall image segmentation scale.

[0081] Device Embodiment

[0082] According to an embodiment of the present invention, an image segmentation device is provided. Figure 3 is a schematic diagram of an image segmentation device according to an embodiment of the present invention. Figure 3 As shown, the image segmentation device according to the embodiment of the present invention specifically includes:

[0083] The acquisition module 30 is used to acquire the image to be segmented and the user's operation prompt on the image to be segmented, specifically for:

[0084] Obtain an image to be segmented after resolution preprocessing;

[0085] The acquisition includes click information of clicking on the region of interest of the image, selection box information of marking the target with a matrix box, and vocabulary information used to describe the target as operation prompts.

[0086] The encoder module 32 is used to input the image to be segmented and the operation prompt into the encoder group, and generate image features and prompt features respectively through the encoder group, specifically for:

[0087] Generate an embedded representation of the image to be segmented through an image encoder, perform a convolution operation on the embedded representation, and generate corresponding image features;

[0088] A two-dimensional prompt map of click information, a binary mask map of selection box information, and a text embedding of vocabulary information are generated through a prompt encoder, and the two-dimensional prompt map, binary mask map and text embedding are used together as prompt features.

[0089] The classification module 34 is used to obtain image features and prompt features through the classification module, extract vocabulary information in the prompt features, use the classification module to output the probability that the image features belong to the vocabulary information category, and when the probability is lower than the threshold, call the corresponding knowledge graph information as an auxiliary feature, specifically for:

[0090] The image features are input into the output layer of the classification module, wherein the classification module is a machine learning model that outputs the probability distribution of each category, indexes the vocabulary information in the probability distribution, and obtains the probability corresponding to the vocabulary information category.

[0091] When the probability is lower than a threshold, a pre-established knowledge graph including vocabulary information is obtained, neighbor nodes with the vocabulary information as the center are obtained from the knowledge graph, and a preset number of neighbor nodes are randomly selected as auxiliary features.

[0092] The decoder module 36 is used to obtain image features, prompt features and auxiliary features through the segmentation interpreter, and combine them into fused features, and output a segmentation mask through the fused features to obtain a segmentation result, specifically for:

[0093] Combine the prompt features and auxiliary features into the total features;

[0094] The cross-attention mechanism is used to perform bidirectional interaction between image features and total features to obtain attention image features and attention total features, which are then concatenated into fusion features.

[0095] Based on the fusion features, the segmentation mask of each pixel in the image to be segmented is output, and the segmentation result is output according to the category of the segmentation mask.

[0096] To sum up, in response to the existing problems, this invention proposes an image segmentation device, which is implemented based on the SAM model; vocabulary information is used as text prompts to eliminate a large amount of redundancy and modifications contained in the original long sentence, and reduce the computational cost of long sentence segmentation using a natural language processing model; on the basis of the original SAM segmentation model, a custom classification module is introduced to pre-evaluate whether the object can be divided in the image based on the vocabulary information. If not, the knowledge graph information is used as an auxiliary feature to assist in inferring the position of the object in the image, so as to avoid the problem that a single vocabulary information is not comprehensive enough. The knowledge graph information is more accurate than the modifications in the original long sentence, which can effectively alleviate the problem of weak generalization ability of the model; since the encoder has obtained the image features, only the output layer is used for prediction within the classification module to minimize the impact of the classification module on the overall image segmentation scale.

[0097] Electronic device embodiment

[0098] Figure 4 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 400 may include at least one processor 410 and a memory 420. The processor 410 may execute instructions stored in the memory 420. The processor 410 is connected to the memory 420 through a data bus. In addition to the memory 420, the processor 410 may also be connected to an input device 430, an output device 440, and a communication device 450 through a data bus.

[0099] The processor 410 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0100] Memory 420 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0101] In the embodiment of the present disclosure, executable instructions are stored in the memory 420, and the processor 410 can read the executable instructions from the memory 420 and execute the instructions to implement all or part of the steps of any image segmentation method in the above exemplary embodiments.

[0102] Computer Readable Storage Medium Embodiments

[0103] In addition to the above-mentioned methods and apparatus, an exemplary embodiment of the present disclosure may also be a computer program product or a computer-readable storage medium storing the computer program product, wherein the computer product includes computer program instructions that can be executed by a processor to implement all or part of the steps described in any of the image segmentation methods in the above-mentioned exemplary embodiments.

[0104] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages ​​and scripting languages ​​(e.g., Python). The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0105] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: a static random access memory (SRAM) with one or more wires electrically connected, an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk, or any suitable combination of the above.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image segmentation method, characterized in that: The method is based on the SAM segmentation model and includes: Acquire an image to be segmented and a user's operation prompt on the image to be segmented; Inputting the image to be segmented and the operation prompt into an encoder group, and generating image features and prompt features respectively through the encoder group; The image feature and the prompt feature are obtained through a classification module, the vocabulary information in the prompt feature is extracted, and the probability that the image feature belongs to the vocabulary information category is outputted using the classification module, and when the probability is lower than a threshold, the corresponding knowledge graph information is called as an auxiliary feature; The image features, the prompt features and the auxiliary features are acquired through a segmentation interpreter and combined into fused features, and a segmentation mask is output through the fused features to obtain a segmentation result.

2. The method according to claim 1, characterized in that The obtaining of the image to be segmented and the user's operation prompt on the image to be segmented specifically include: Obtain an image to be segmented after resolution preprocessing; The acquisition includes click information of clicking on the region of interest of the image, selection box information of marking the target with a matrix box, and vocabulary information used to describe the target as operation prompts.

3. The method according to claim 2, characterized in that The generating of the image features and the prompt features by the encoder group specifically includes: Generate an embedded representation of the image to be segmented by an image encoder, and perform a convolution operation on the embedded representation to generate corresponding image features; A two-dimensional prompt map of the click information is generated by a prompt encoder, a binary mask map of the selection box information is generated, and a text embedding of the vocabulary information is generated. The two-dimensional prompt map, the binary mask map and the text embedding are used together as prompt features.

4. The method according to claim 1, characterized in that The using the classification module to output the probability that the image feature belongs to the vocabulary information category specifically includes: The image features are input into the output layer of a classification module, wherein the classification module is a machine learning model that outputs a probability distribution of each category, indexes vocabulary information in the probability distribution, and obtains the probability corresponding to the vocabulary information category.

5. The method according to claim 1, characterized in that When the probability is lower than the threshold, calling the corresponding knowledge graph information as an auxiliary feature specifically includes: When the probability is lower than a threshold, a pre-established knowledge graph including vocabulary information is obtained, neighbor nodes with the vocabulary information as the center are obtained from the knowledge graph, and a preset number of neighbor nodes are randomly selected as auxiliary features.

6. The method according to claim 1, characterized in that The step of obtaining the image features, the prompt features and the auxiliary features through a segmentation interpreter and combining them into fusion features specifically includes: splicing the prompt feature and the auxiliary feature into a total feature; A cross-attention mechanism is used to perform bidirectional interaction between the image features and the total features to obtain attention image features and attention total features, and the attention image features and the attention total features are spliced ​​into a fusion feature.

7. The method according to claim 1, characterized in that The step of outputting a segmentation mask through the fusion feature to obtain a segmentation result specifically includes: A segmentation mask for each pixel in the image to be segmented is output based on the fusion feature, and a segmentation result is output according to the category of the segmentation mask.

8. An image segmentation device, characterized in that: include: An acquisition module, used for acquiring the image to be segmented and the operation prompt of the user on the image to be segmented; An encoder module, used for inputting the image to be segmented and the operation prompt into an encoder group, and generating image features and prompt features respectively through the encoder group; A classification module, used to obtain the image features and the prompt features through the classification module, extract the vocabulary information in the prompt features, use the classification module to output the probability that the image features belong to the vocabulary information category, and when the probability is lower than a threshold, call the corresponding knowledge graph information as an auxiliary feature; The decoder module is used to obtain the image features, the prompt features and the auxiliary features through the segmentation interpreter, and combine them into fused features, and output the segmentation mask through the fused features to obtain the segmentation result.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the image segmentation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 7 are implemented.

Images