Image segmentation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image segmentation method and device, electronic equipment and a readable storage medium. The method comprises the following steps: receiving a to-be-segmented image and text information describing the to-be-segmented image; based on the to-be-segmented image and the text information, determining an image feature, an image embedding feature and a text embedding feature corresponding to the to-be-segmented image; processing the image features, the image embedding features and the text embedding features through a decoder module based on cross attention and a multi-layer perceptron to obtain a first image vector and a first text vector, and obtaining a first fusion vector based on the first image vector and the first text vector; obtaining an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in the segmentation decoder; and segmenting the to-be-segmented image based on the image segmentation result to obtain a target area in the to-be-segmented image. Through the image segmentation method, the segmentation performance of image segmentation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image segmentation method, device, electronic device and readable storage medium. Background Art

[0002] Image segmentation is widely used in intelligent driving, video analysis, remote sensing detection and other fields. In the existing technology, tasks such as semantic segmentation, instance segmentation and full-view segmentation can be achieved through encoder-decoder architecture, dilated convolution, pyramid pooling and other methods.

[0003] However, most methods for implementing image segmentation rely on explicit human instructions or predefined categories to identify target objects, lacking the ability to implicitly reason autonomously and understand user intentions, limiting the application of intelligent perception systems in more natural and intuitive human-computer interaction scenarios. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide an image segmentation method, device, electronic device and readable storage medium to solve the problem of lack of implicit autonomous reasoning and the ability to understand user intentions in the prior art.

[0005] According to a first aspect of the embodiments of the present disclosure, there is provided an image segmentation method, comprising:

[0006] Receiving an image to be segmented and text information describing the image to be segmented;

[0007] Based on the image to be segmented and the text information, determining the image features, image embedding features and text embedding features corresponding to the image to be segmented;

[0008] By means of a decoder module in a segmentation decoder, the image feature, the image embedding feature and the text embedding feature are processed based on a cross attention and a multi-layer perceptron to obtain a first image vector and a first text vector, and a first fusion vector is obtained based on the first image vector and the first text vector;

[0009] Obtaining an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in the segmentation decoder;

[0010] The image to be segmented is segmented based on the image segmentation result to obtain the target area in the image to be segmented.

[0011] According to a second aspect of the embodiments of the present disclosure, there is provided an image segmentation device, comprising:

[0012] A receiving module, configured to receive an image to be segmented and text information describing the image to be segmented;

[0013] A first processing module is configured to determine image features, image embedding features and text embedding features corresponding to the image to be segmented based on the image to be segmented and text information;

[0014] A second processing module is configured to process the image features, the image embedding features and the text embedding features based on cross attention and a multi-layer perceptron through a decoder module in the segmentation decoder to obtain a first image vector and a first text vector, and obtain a first fusion vector based on the first image vector and the first text vector;

[0015] A third processing module is configured to obtain an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in the segmentation decoder;

[0016] The segmentation module is configured to segment the image to be segmented based on the image segmentation result to obtain the target area in the image to be segmented.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0019] Compared with the prior art, the disclosed embodiments have the following beneficial effects: receiving an image to be segmented and text information describing the image to be segmented, determining image features, image embedding features and text embedding features corresponding to the image to be segmented according to the image to be segmented and the text information, processing the image features, image embedding features and text embedding features through a decoder module in a segmentation decoder according to cross attention and a multi-layer perceptron to obtain a first image vector and a first text vector, and obtaining a first fusion vector according to the first image vector and the first text vector, so that information can be screened and enhanced within a single modality, obtaining an image segmentation result according to the first image vector, the first text vector and the first fusion vector through a segmentation head module in the segmentation decoder, enabling effective information exchange and fusion between modalities, improving the segmentation decoder's ability to integrate multimodal data, segmenting the image to be segmented according to the image segmentation result to obtain a target area in the image to be segmented, and improving the segmentation performance of image segmentation, thereby promoting the development potential of intelligent perception systems in autonomously understanding user intent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of an image segmentation method provided by an embodiment of the present disclosure;

[0023] Figure 3 is a flowchart of another image segmentation method provided by an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of a segmentation decoder provided by an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of an image segmentation device provided by an embodiment of the present disclosure;

[0026] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0028] An image segmentation method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a server 104, and a network 105.

[0030] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be hardware or software. When the first terminal device 101, the second terminal device 102, and the third terminal device 103 are hardware, they can be various electronic devices with display screens and supporting communication with the server 104, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers; when the first terminal device 101, the second terminal device 102, and the third terminal device 103 are software, they can be installed in the electronic devices described above. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be implemented as multiple software or software modules, or as a single software or software module, which is not limited in the embodiments of the present disclosure. Further, various applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0031] The server 104 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 104 may be a single server, a server cluster consisting of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0032] It should be noted that the server 104 can be hardware or software. When the server 104 is hardware, it can be various electronic devices that provide various services for the first terminal device 101, the second terminal device 102, and the third terminal device 103. When the server 104 is software, it can be multiple software or software modules that provide various services for the first terminal device 101, the second terminal device 102, and the third terminal device 103, or it can be a single software or software module that provides various services for the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the embodiments of the present disclosure are not limited to this.

[0033] The network 105 may be a wired network connected by coaxial cables, twisted pairs, and optical fibers, or a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0034] The user can establish a communication connection with the server 104 via the network 105 through the first terminal device 101, the second terminal device 102 and the third terminal device 103 to receive or send information, etc. Specifically, after the user imports the collected data of the points of interest into the server 104, the server 104 obtains the first data of the points of interest to be processed, which includes the first longitude and latitude and the first classification of the points of interest to be processed, and performs a conflict check on the points of interest to be processed according to the first longitude and latitude and the first classification; further, in the case of determining a conflict, the server 104 performs conflict processing on the points of interest to be processed to avoid a large amount of duplicate data and unavailable data in the database.

[0035] It should be noted that the specific types, quantities and combinations of the first terminal device 101, the second terminal device 102 and the third terminal device 103, the server 104 and the network 105 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure do not limit this.

[0036] In the field of image segmentation, a large number of studies have been conducted in the prior art to perform image segmentation tasks through different design schemes, such as encoder-decoder architecture, dilated convolution, pyramid pooling modules, etc., which have significantly promoted the development of tasks such as semantic segmentation, instance segmentation, and full-view segmentation. However, the prior art still faces a key problem: most of them rely on explicit human instructions or predefined categories to identify target objects in the image to be segmented, and lack the ability to autonomously reason and understand user intentions based on implicit instructions. For example, the existing intelligent perception system may not be able to handle a simple command from the user to "change the channel", and requires more detailed step-by-step instructions. This limitation restricts the application of intelligent perception systems in more natural and intuitive human-computer interaction scenarios, especially in the industrial and robotic fields that require complex reasoning and understanding of world knowledge.

[0037] In response to the above problems, the present disclosure provides a model that can generate accurate binary segmentation masks based on implicit query text. Compared with the existing technology, the image segmentation algorithm not only requires the model to be able to understand complex text descriptions, but also requires the ability to perform deep reasoning in combination with the content of the image to be segmented. In order to achieve this goal, the present disclosure introduces a multimodal large language model LLM (Large Language Model), which inherits the language generation capability of LLM, and obtains the ability to generate segmentation masks by introducing special tags and embeddings (i.e., masks). Therefore, with limited reasoning segmentation data samples, the model parameters can be fine-tuned, which enables the model to exhibit strong zero-sample reasoning capability and segmentation performance, thereby promoting the development potential of intelligent perception systems in autonomously understanding user intentions.

[0038] Among them, zero samples are due to the fact that the amount of training sample data in the multimodal LLM is large enough, and there is no need to retrain when new semantics are received, and there is no need to perform specific learning for a certain content.

[0039] Figure 2 It is a flowchart of an image segmentation method provided by an embodiment of the present disclosure. Figure 2 The segmentation decoder in the image segmentation method can be stored in Figure 1 The terminal device or server in the image segmentation method can be Figure 1 The terminal device or server in the Figure 2 As shown, the image segmentation method includes:

[0040] S201, receiving an image to be segmented and text information describing the image to be segmented;

[0041] Specifically, an image to be segmented and text information describing the image to be segmented are received, wherein characteristics of the image to be segmented, such as image quality, resolution, and color depth, are recorded, and the text information can be in the form of a short description of the image to be segmented, a label set, or a more detailed natural language description, etc., which can provide additional contextual information about the image content of the image to be segmented.

[0042] The source of the text information may be manually input text information, text information extracted from a database, etc., which can enable the segmentation decoder to understand the image content more accurately, thereby guiding the segmentation process and obtaining a more accurate image segmentation result.

[0043] Furthermore, the image to be segmented can be preprocessed, that is, the size of the image to be segmented, color space conversion, denoising, etc. can be adjusted, and the process of preprocessing the text information can be natural language processing steps such as word segmentation, stop word removal, stem extraction, word frequency statistics, etc., so as to extract the key information of image segmentation from the text information.

[0044] Receiving the image to be segmented and the corresponding text information enables the segmentation decoder to obtain multi-source data and fuse multiple data, thereby performing image segmentation on the image to be segmented more accurately.

[0045] S202, determining image features, image embedding features, and text embedding features corresponding to the image to be segmented based on the image to be segmented and the text information;

[0046] Specifically, according to the image to be segmented and the text information, the image features of the image to be segmented can be extracted by image feature extraction, and the image to be segmented is input into a preset multimodal LLM model to extract the image embedding features of the image to be segmented. At the same time, the text information is input into the preset multimodal LLM model to obtain the text embedding features.

[0047] Among them, image features can represent basic information such as color, texture, shape, etc. in the image to be segmented, image embedding features are used to express the intrinsic information of the image to be segmented, and text embedding features are used to represent the position of a word or phrase in the text in the semantic space. The data forms of image features, image embedding features, and text embedding features are all in vector form.

[0048] Furthermore, image features can be extracted through image encoders. Image features are the basis of image segmentation and provide rich visual information for subsequent multimodal fusion. Image embedding features and text embedding features can be extracted through the preset multimodal LLM model, which can enhance the preset multimodal LLM model's ability to understand and integrate multimodal information. Among them, the preset multimodal LLM model can be fine-tuned through low-rank adaptation LoRA (Low-Rank Adaptation), so that the preset multimodal LLM model reduces the parameters that need to be adjusted and improves computational efficiency.

[0049] S203, processing the image features, the image embedding features, and the text embedding features based on cross attention and a multi-layer perceptron by a decoder module in the segmentation decoder to obtain a first image vector and a first text vector, and obtaining a first fusion vector based on the first image vector and the first text vector;

[0050] Specifically, the image features, image embedding features and text features are further processed through the decoder module in the segmentation decoder. The processing process can enable the image features to focus on the image parts related to the text information through the cross-attention mechanism. Conversely, the text embedding features can also better focus on the parts related to the image features, which can enhance the understanding of the image content.

[0051] At the same time, the image features, image embedding features, text embedding features, etc. are transformed nonlinearly through the multilayer perceptron (MLP) to extract deeper feature representations. MLP is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0052] In addition, through cross attention and multi-layer perceptron, a first image vector and a first text vector can be obtained, and the first image vector and the first text vector are processed to obtain a first fusion vector, which retains the characteristics of the first image vector and the first text vector, wherein the processing method can be through a gated fusion mechanism.

[0053] S204, obtaining an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in the segmentation decoder;

[0054] Specifically, the first image vector, the first text vector and the first fusion vector are input into a segmentation head module in a segmentation decoder to obtain an image segmentation result.

[0055] Among them, in the processing process of the segmentation head module, convolution, multi-layer perceptron and other methods may be involved.

[0056] S205, segmenting the image to be segmented based on the image segmentation result to obtain a target area in the image to be segmented.

[0057] Specifically, the image to be segmented is segmented according to the image segmentation result to obtain a target area in the image to be segmented. The target area can be identified according to different objects, scenes or specific features in the image. For example, in the application scenario of autonomous driving, the target area can be traffic elements such as roads, vehicles, and pedestrians.

[0058] Furthermore, when there are different target regions in the image to be segmented, different target regions may be marked differently, such as with different colors, contour lines, and the like.

[0059] Figure 3 FIG. 1 is a flow chart of another image segmentation method provided by an embodiment of the present disclosure. Figure 3 As shown, the image to be segmented is input into the image encoder and the preset multimodal LLM model respectively, the image features are obtained through the image encoder, and the image features are input into the segmentation decoder. At the same time, the text information is input into the preset multimodal LLM model to obtain image embedding features and text embedding features, and the image embedding features and text embedding features are also input into the segmentation decoder. The image features, image embedding features and text embedding features are processed by the segmentation decoder to obtain the image segmentation result.

[0060] According to the technical solution provided by the embodiments of the present disclosure, image features, image embedding features and text embedding features can be determined based on the image to be segmented and the corresponding text information, and the image features, image embedding features and text embedding features can be processed to obtain image segmentation results, thereby completing the segmentation of the image to be segmented and improving the segmentation performance of image segmentation.

[0061] In some embodiments, based on the image to be segmented and the text information, determining the image features, image embedding features and text embedding features corresponding to the image to be segmented includes:

[0062] Based on the image encoder, image features are extracted from the image to be segmented;

[0063] The image to be segmented and the text information are input into a preset multimodal large language model to obtain the image embedding features and text embedding features corresponding to the image to be segmented.

[0064] Specifically, the image features of the image to be segmented are extracted through the image encoder. The image encoder can understand the image content of the image to be segmented, which includes color, shape, texture, and more complex structural information, thereby converting these visual information into numerical features that can be understood by a computer. Image features are high-dimensional, dense vector representations that can capture the main elements and details in the image.

[0065] In addition, the image and text information to be segmented are processed by a preset multimodal LLM model to obtain the corresponding image embedding features and text embedding features.

[0066] Among them, the preset multimodal LLM model has the ability to process multiple types of data, can understand and analyze image and text information at the same time, and realize cross-modal information fusion. After input, the image embedding features and text embedding features corresponding to the image to be segmented are extracted through the neural network architecture.

[0067] Among them, the image embedding feature can represent the image to be segmented compactly and rich in semantic information, reflecting the category, positional relationship and contextual environment of the objects in the image; the text embedding feature converts the text information into a representation form compatible with the image feature space, so that the description, label or instruction in the text information can effectively interact and match with the image features.

[0068] According to the technical solution provided by the embodiments of the present disclosure, the image to be segmented and the text information can be deeply combined according to the preset multimodal LLM model, and the determined image embedding features and text embedding features can represent the potential association and correspondence between the image to be segmented and the text information. A cross-modal feature extraction method enhances the preset multimodal LLM model's ability to understand the image content, and can also more accurately determine the image segmentation result based on the text information.

[0069] In some embodiments, the decoder module in the segmentation decoder processes the image features, the image embedding features, and the text embedding features based on cross attention and a multi-layer perceptron to obtain a first image vector and a first text vector, including:

[0070] Perform self-attention processing on the image embedding features to obtain a first query vector, and use the image features as the first key vector and the first value vector to perform cross-attention processing to obtain an initial image vector;

[0071] Processing the initial image vector through a first multilayer perceptron to obtain a first image vector;

[0072] Perform self-attention processing on the text embedding features to obtain the second query vector, and use the image features as the second key vector and the second value vector to perform cross-attention processing to obtain the initial text vector;

[0073] The initial text vector is processed by a second multi-layer perceptron to obtain a first text vector.

[0074] Specifically, the segmentation decoder includes a decoder module and a segmentation head module. The decoder module can process the image features, image embedding features and text embedding features through cross attention and MLP to obtain the first image vector and the first text vector. The process is:

[0075] For image embedding features, the decoder module performs self-attention processing on the image embedding features, which can determine the correlation between the elements in the image embedding features and generate a first query vector. The first query vector can play a guiding role in the cross-attention processing and help the decoder module focus on the most critical information in the image to be segmented.

[0076] In addition, the image features of the image to be segmented are used as the first key vector and the first vector, and cross-attention processing is performed with the first query vector to obtain an initial image vector. The initial image vector integrates the semantic information of the image embedded features, and also establishes a connection with the image feature features of the image to be segmented through cross-attention, thereby enhancing the expression ability.

[0077] In addition, the initial image vector is processed by the first multilayer perceptron (MLP1), which can learn the nonlinear mapping relationship, further abstract and extract features of the initial image vector, and generate the first image vector. The first image vector retains the key information of the image to be segmented, and also enhances the ability to distinguish different target areas through the transformation of MLP.

[0078] In addition, similarly, for the text embedding features, the decoder module performs self-attention processing on them to obtain a second query vector. The second query vector can extract key elements and contextual information in the text information, and use the image features as the second key vector and the second value vector, and perform cross-attention processing with the second query vector to obtain an initial text vector. The initial text vector realizes the cross-modal fusion of text information and image features, contains the semantic content of the text information, and also establishes associations with image features through cross-attention.

[0079] In addition, in order to optimize the representation of the initial text vector, the decoder module processes the initial text vector through the second multi-layer perceptron MPL2. Through multiple fully connected layers and nonlinear transformations, MLP2 can further extract and integrate key information in the initial text vector to generate the first text vector. Among them, the first text vector has a high degree of semantic expression ability and can also work together with the first image vector to jointly guide the image segmentation task.

[0080] According to the technical solution provided in the embodiment of the present disclosure, by segmenting the decoder module in the decoder, cross-attention and multi-layer perceptron are used to process the image features, image embedding features and text embedding features to obtain a first image vector and a first text vector, which can integrate the key contents of the image and text information, and enhance its representation ability and discrimination through the cross-attention mechanism and MLP transformation.

[0081] In some embodiments, obtaining a first fusion vector based on the first image vector and the first text vector includes:

[0082] The image feature is used as a third query vector, and the first image feature is used as a third key vector and a third value vector, and a cross attention process is performed to obtain a second image vector;

[0083] The image feature is used as a fourth query vector, and the first text feature is used as a fourth key vector and a fourth value vector, and a cross attention process is performed to obtain a second text vector;

[0084] The second image vector and the second text vector are fused through a gated fusion mechanism to obtain a first fused vector.

[0085] Specifically, the image feature is used as the third query vector, which can be understood as a "probe" that actively searches for information during the search or matching process. At the same time, the first image feature is used as the third key vector and the third value vector, which can be understood as playing the roles of "index" and "content" respectively, and interacting with the query vector to extract relevant information. Through cross-attention processing, the image feature can self-reinforce or adjust according to its own key information (provided by the first image feature) to generate a second image vector, thereby enhancing the internal coherence of the image feature and the richness of information.

[0086] At the same time, the image feature is used as the fourth query vector, and the first text feature (the preliminarily extracted text feature) is used as the fourth key vector and the fourth value vector, so that the image feature can "interrogate" the text feature. Through their interaction, the image feature can obtain the contextual information or semantic content related to it in the text, so as to form a cross-modal information exchange and generate a second text vector containing richer contextual information.

[0087] In addition, the second image vector and the second text vector are integrated through a gated fusion mechanism to obtain a first fused vector. The gated fusion mechanism is a method for dynamically adjusting the inflow and outflow of information, and can determine the information to be retained, enhanced, or suppressed based on the characteristics of the second image vector and the second text vector. The gated fusion mechanism ensures that the fusion process is both efficient and flexible, and can maximize the retention and utilization of useful information in the two vectors while reducing unnecessary noise or redundancy.

[0088] According to the technical solution provided in the embodiment of the present disclosure, it is possible to obtain a more comprehensive and accurate first fusion vector that fuses both image and text information based on the first image vector and the second text vector through cross-attention processing and gated fusion mechanism.

[0089] In some embodiments, obtaining an image segmentation result based on the first image vector, the first text vector and the first fusion vector by a segmentation head module in the segmentation decoder includes:

[0090] The first image vector and the first text vector are fused through a gated fusion mechanism to obtain a second fused vector;

[0091] Based on the first fusion vector and the second fusion vector, obtaining at least one segmentation mask result and an average intersection-over-union ratio corresponding to each segmentation mask result, where the average intersection-over-union ratio is used to indicate a similarity between the corresponding segmentation mask result and a preset segmentation result;

[0092] The average intersection-union ratio is determined as the weight of the corresponding segmentation mask result, and a weighted average operation is performed on all segmentation mask results to obtain the image segmentation result.

[0093] Specifically, in the segmentation head module, the first image vector and the first text vector are fused through a gated fusion mechanism to generate a second fused vector, thereby combining the features of the image and the text, so that the fused second fused vector can capture the complementary information between the two modalities.

[0094] Among them, the gated fusion mechanism can effectively integrate useful information by dynamically adjusting the weight of the information flow, thereby generating a second fusion vector that contains the image details of the image to be segmented and the context information in the text information.

[0095] In addition, at least one segmentation mask result is generated according to the first fusion vector and the second fusion vector through the operation in the segmentation head module, wherein each segmentation mask result is a possible representation of dividing different areas in the image to be segmented, and each pixel point is assigned a category label.

[0096] At the same time, the average intersection over union (IoU) corresponding to each segmentation mask result is determined by the segmentation head module, and the average intersection over union (IoU) can be used as a pixel-level weight.

[0097] Among them, the average intersection over union ratio is an evaluation indicator used to quantify the similarity between the segmentation mask result and the preset segmentation result (i.e., the true label or reference segmentation). That is, IoU determines the average value of the ratio of the intersection area between the predicted segmentation area and the true segmentation area to the union area. The higher the average value, the more accurate the prediction result.

[0098] In addition, the average IoU of each segmentation mask result is used as the weight of the segmentation mask result, and a weighted average operation is performed on all segmentation mask results, so that at least one segmentation result is merged into a more reliable image segmentation result through the reliable performance indicator provided by IoU.

[0099] Among them, the weighted average operation can make the segmentation mask result that is closer to the true segmentation occupy a larger proportion in the image segmentation result, thereby improving the accuracy and robustness of image segmentation.

[0100] According to the technical solution provided by the embodiment of the present disclosure, through the segmentation head module in the segmentation decoder, at least one segmentation mask result can be generated through multiple feature vectors, and the average intersection-and-union ratio can be obtained. The segmentation mask result is weighted averaged according to the average intersection-and-union ratio, so as to obtain an accurate and stable image segmentation result.

[0101] In some embodiments, obtaining at least one segmentation mask result and an average intersection-over-union ratio corresponding to each segmentation mask result based on the first fusion vector and the second fusion vector includes:

[0102] Processing the second fusion vector through a third multi-layer perceptron to obtain a perception vector, where the perception vector is used to represent an initial segmentation mask result;

[0103] Performing convolution processing on the first fusion vector, and adding the convolved first fusion vector and the perception vector to obtain a first target fusion vector, where the first target fusion vector corresponds to at least one segmentation mask result;

[0104] The second fusion vector is processed by a fourth multilayer perceptron to obtain a second target fusion vector, the dimension of the second target fusion vector is the same as the dimension of the first target fusion vector, and the second target fusion vector corresponds to at least one average intersection-and-union ratio, and the at least one average intersection-and-union ratio corresponds to at least one segmentation mask result.

[0105] Specifically, the second fused vector is processed by the third multi-layer perceptron MLP3, and a perception vector is obtained through nonlinear transformation of MLP3. The perception vector is used as an initial segmentation mask result, that is, an intermediate representation of the segmentation mask result, including information required for generating the segmentation mask. Among them, MLP3 can extract key features from the second fused vector and can characterize potential segmentation mask results.

[0106] In addition, a convolution process is performed on the first fusion vector, and the convolved first fusion vector is added to the obtained perception vector to obtain a first target fusion vector. The convolution operation can capture local features and enhance the correlation between features. The first target fusion vector contains spatial information from the first fusion vector and also incorporates high-level semantic information from the second fusion vector, so that it can correspond to at least one segmentation mask result.

[0107] In addition, in order to obtain the average intersection-over-union (IoU) corresponding to these segmentation mask results, the second fused vector can be processed by another head in the segmentation head (ie, another processing path), that is, the second fused vector is processed by the fourth multilayer perceptron MLP4.

[0108] The second target fusion vector is obtained through the nonlinear transformation of MLP4. The design purpose of MLP4 is different from that of MLP3, which is used to extract features for evaluating the segmentation quality (i.e., calculating IoU) from the second fusion vector.

[0109] The dimension of the second target fusion vector is the same as that of the first target fusion vector, so that the segmentation mask result and the intersection-over-union ratio are one-to-one corresponding, and thus can be effectively calculated. The average intersection-over-union ratio is used as an evaluation index of segmentation performance, which can quantify the similarity between the segmentation mask result and the true segmentation, thereby providing the corresponding segmentation quality.

[0110] According to the technical solution provided in the embodiment of the present disclosure, at least one segmentation mask result and an average intersection-over-union ratio corresponding to each segmentation mask result can be generated from the first fusion vector and the second fusion vector through the processing of the third multilayer perceptron and the fourth multilayer perceptron and the convolution operation on the first fusion vector, thereby improving the refinement of the image segmentation result.

[0111] Figure 4 is a schematic diagram of a segmentation decoder provided by an embodiment of the present disclosure. Figure 4 As shown, the segmentation decoder includes a decoder module and a segmentation head module. The decoder module includes two branches, which are used to process image embedding features and text embedding features respectively.

[0112] In the image embedding feature branch: the image embedding feature is input into the decoder module, the image embedding feature is processed by self-attention to obtain the first query vector, and the image feature is used as the first key vector and the first value vector, and cross-attention processing is performed to obtain the initial image vector, the initial image vector is processed by the first multi-layer perceptron MLP1 to obtain the first image vector, and then cross-attention processing is performed together with the image feature to obtain the second image vector.

[0113] In the text embedding feature branch: perform self-attention processing on the text embedding features to obtain the second query vector, and use the image features as the second key vector and the second value vector for cross-attention processing to obtain the initial text vector, and process the initial text vector through the second multi-layer perceptron MLP2 to obtain the first text vector, and then perform cross-attention processing with the image features to obtain the second text vector.

[0114] After obtaining the second image vector and the second text vector, they are fused through a gated fusion mechanism to obtain a first fused vector.

[0115] In the segmentation head module, the first image vector and the first text vector are fused through a gated fusion mechanism to obtain a second fused vector, the second fused vector is processed through a third multilayer perceptron to obtain a perception vector, convolution processing is performed on the first fusion vector, and the convolved first fusion vector and the perception vector are added to obtain a first target fusion vector, the second fusion vector is processed through a fourth multilayer perceptron to obtain a second target fusion vector, the second target fusion vector corresponds to at least one average intersection-and-union ratio, the average intersection-and-union ratio is determined as the weight of the corresponding segmentation mask result, and a weighted average operation is performed on all segmentation mask results to obtain an image segmentation result.

[0116] In some embodiments, after segmenting the image to be segmented based on the image segmentation result to obtain the target area in the image to be segmented, the method further includes:

[0117] Obtaining decoding parameters corresponding to the decoder module and segmentation parameters corresponding to the segmentation header module, and obtaining a preset segmentation result;

[0118] Input the image segmentation result and the preset segmentation result into the preset loss function to determine the corresponding loss value;

[0119] Based on the loss value, the decoding parameters and segmentation parameters are adjusted.

[0120] Specifically, after completing the segmentation of the target area in the image to be segmented, the decoding parameters corresponding to the decoder module and the segmentation parameters corresponding to the segmentation head module are obtained.

[0121] Among them, the decoding parameters are used to decode the encoded feature map back to the original image space, and the segmentation parameters are used to generate an image segmentation result based on the decoded feature map.

[0122] At the same time, a preset segmentation result is obtained. The preset segmentation result can be a highly accurate segmentation result annotated by an expert, or a segmentation result obtained by other high-precision methods. It is used as a reference standard to evaluate the performance of the current image segmentation algorithm.

[0123] In addition, the image segmentation result and the preset segmentation result are input into a preset loss function, which can be a function that measures the difference between the two segmentation results. Different types can be selected as needed, such as cross entropy loss, Dice loss, etc. The loss value calculated by the preset loss function can quantitatively evaluate the degree of difference between the current image segmentation algorithm and the preset segmentation result.

[0124] In addition, after obtaining the loss value, the decoding parameters and segmentation parameters can be adjusted according to the loss value, which can be achieved through the back propagation algorithm, that is, the gradient is calculated according to the loss value, and the parameter value is updated according to the direction of the gradient. By iterating the above process multiple times, the loss value can be gradually reduced, thereby improving the performance of the image segmentation algorithm.

[0125] Furthermore, in the process of adjusting parameters, we need to pay attention to the problems of overfitting and underfitting to ensure good performance on both the training set and the test set.

[0126] According to the technical solution provided by the embodiment of the present disclosure, after the image to be segmented is segmented according to the image segmentation result and the target area is obtained, a series of steps such as obtaining decoding parameters and segmentation parameters, obtaining preset segmentation results, calculating loss values ​​and adjusting parameters are required to continuously optimize the segmentation model. These steps together constitute a complete image segmentation process, which provides strong support for image segmentation tasks.

[0127] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0128] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0129] Figure 5 Schematic diagram of an image segmentation device provided by an embodiment of the present disclosure. Figure 5 As shown, the image segmentation device includes: a receiving module 501, a first processing module 502, a second processing module 503, a third processing module 504, and a segmentation module 505, wherein:

[0130] A receiving module 501 is configured to receive an image to be segmented and text information describing the image to be segmented;

[0131] The first processing module 502 is configured to determine the image features, image embedding features and text embedding features corresponding to the image to be segmented based on the image to be segmented and the text information;

[0132] The second processing module 503 is configured to process the image features, the image embedding features and the text embedding features through the decoder module in the segmentation decoder based on the cross attention and the multi-layer perceptron to obtain a first image vector and a first text vector, and obtain a first fusion vector based on the first image vector and the first text vector;

[0133] A third processing module 504 is configured to obtain an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in the segmentation decoder;

[0134] The segmentation module 505 is configured to segment the image to be segmented based on the image segmentation result to obtain a target area in the image to be segmented.

[0135] In some embodiments, the first processing module 502 is configured to:

[0136] Based on the image encoder, image features are extracted from the image to be segmented;

[0137] The image to be segmented and the text information are input into a preset multimodal large language model to obtain the image embedding features and text embedding features corresponding to the image to be segmented.

[0138] In some embodiments, the second processing module 503 is configured to:

[0139] Perform self-attention processing on the image embedding features to obtain a first query vector, and use the image features as the first key vector and the first value vector to perform cross-attention processing to obtain an initial image vector;

[0140] Processing the initial image vector through a first multilayer perceptron to obtain a first image vector;

[0141] Perform self-attention processing on the text embedding features to obtain the second query vector, and use the image features as the second key vector and the second value vector to perform cross-attention processing to obtain the initial text vector;

[0142] The initial text vector is processed by a second multi-layer perceptron to obtain a first text vector.

[0143] In some embodiments, the second processing module 503 is configured to:

[0144] The image feature is used as a third query vector, and the first image feature is used as a third key vector and a third value vector, and a cross attention process is performed to obtain a second image vector;

[0145] The image feature is used as a fourth query vector, and the first text feature is used as a fourth key vector and a fourth value vector, and a cross attention process is performed to obtain a second text vector;

[0146] The second image vector and the second text vector are fused through a gated fusion mechanism to obtain a first fused vector.

[0147] In some embodiments, the third processing module 504 is configured to:

[0148] The first image vector and the first text vector are fused through a gated fusion mechanism to obtain a second fused vector;

[0149] Based on the first fusion vector and the second fusion vector, obtaining at least one segmentation mask result and an average intersection-over-union ratio corresponding to each segmentation mask result, where the average intersection-over-union ratio is used to indicate a similarity between the corresponding segmentation mask result and a preset segmentation result;

[0150] The average intersection-union ratio is determined as the weight of the corresponding segmentation mask result, and a weighted average operation is performed on all segmentation mask results to obtain the image segmentation result.

[0151] In some embodiments, the third processing module 504 is configured to:

[0152] Processing the second fusion vector through a third multi-layer perceptron to obtain a perception vector, where the perception vector is used to represent an initial segmentation mask result;

[0153] Performing convolution processing on the first fusion vector, and adding the convolved first fusion vector and the perception vector to obtain a first target fusion vector, where the first target fusion vector corresponds to at least one segmentation mask result;

[0154] The second fusion vector is processed by a fourth multilayer perceptron to obtain a second target fusion vector, the dimension of the second target fusion vector is the same as the dimension of the first target fusion vector, and the second target fusion vector corresponds to at least one average intersection-and-union ratio, and the at least one average intersection-and-union ratio corresponds to at least one segmentation mask result.

[0155] In some embodiments, after the segmentation module 505 segments the image to be segmented based on the image segmentation result and obtains the target area in the image to be segmented, it is further configured to:

[0156] Obtaining decoding parameters corresponding to the decoder module and segmentation parameters corresponding to the segmentation header module, and obtaining a preset segmentation result;

[0157] Input the image segmentation result and the preset segmentation result into the preset loss function to determine the corresponding loss value;

[0158] Based on the loss value, the decoding parameters and segmentation parameters are adjusted.

[0159] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0160] Figure 6 Schematic diagram of an electronic device 6 provided in an embodiment of the present disclosure. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0161] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.

[0162] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0163] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0164] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0165] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0166] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An image segmentation method, characterized in that: include: Receiving an image to be segmented and text information describing the image to be segmented; Based on the image to be segmented and the text information, determining image features, image embedding features and text embedding features corresponding to the image to be segmented; Processing the image features, the image embedding features, and the text embedding features based on cross attention and a multi-layer perceptron through a decoder module in a segmentation decoder to obtain a first image vector and a first text vector, and obtaining a first fusion vector based on the first image vector and the first text vector; Obtaining an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in a segmentation decoder; The image to be segmented is segmented based on the image segmentation result to obtain a target area in the image to be segmented.

2. The method according to claim 1, characterized in that Based on the image to be segmented and the text information, determining image features, image embedding features, and text embedding features corresponding to the image to be segmented includes: Extracting image features from the image to be segmented based on an image encoder; The image to be segmented and the text information are input into a preset multimodal large language model to obtain image embedding features and text embedding features corresponding to the image to be segmented.

3. The method according to claim 1, characterized in that The image feature, the image embedding feature and the text embedding feature are processed by a decoder module in a segmentation decoder based on a cross attention and a multi-layer perceptron to obtain a first image vector and a first text vector, including: Performing self-attention processing on the image embedding feature to obtain a first query vector, and performing cross-attention processing on the image feature as a first key vector and a first value vector to obtain an initial image vector; Processing the initial image vector through a first multilayer perceptron to obtain a first image vector; Performing self-attention processing on the text embedding feature to obtain a second query vector, and performing cross-attention processing on the image feature as a second key vector and a second value vector to obtain an initial text vector; The initial text vector is processed by a second multilayer perceptron to obtain a first text vector.

4. The method according to claim 1, characterized in that: Obtaining a first fusion vector based on the first image vector and the first text vector includes: Using the image feature as a third query vector, using the first image feature as a third key vector and a third value vector, performing cross attention processing, and obtaining a second image vector; Taking the image feature as a fourth query vector, taking the first text feature as a fourth key vector and a fourth value vector, performing cross attention processing, and obtaining a second text vector; The second image vector and the second text vector are fused through a gated fusion mechanism to obtain the first fused vector.

5. The method according to claim 1, characterized in that Obtaining an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in a segmentation decoder, including: The first image vector and the first text vector are fused by a gated fusion mechanism to obtain a second fused vector; Based on the first fusion vector and the second fusion vector, obtaining at least one segmentation mask result and an average intersection-over-union ratio corresponding to each segmentation mask result, wherein the average intersection-over-union ratio is used to indicate a similarity between the corresponding segmentation mask result and a preset segmentation result; The average intersection-over-union ratio is determined as the weight of the corresponding segmentation mask result, and a weighted average operation is performed on all the segmentation mask results to obtain an image segmentation result.

6. The method according to claim 5, characterized in that Based on the first fusion vector and the second fusion vector, obtaining at least one segmentation mask result and an average intersection-over-union ratio corresponding to each segmentation mask result, including: Processing the second fusion vector through a third multi-layer perceptron to obtain a perception vector, wherein the perception vector is used to represent an initial segmentation mask result; Performing convolution processing on the first fusion vector, and adding the convolved first fusion vector and the perception vector to obtain a first target fusion vector, where the first target fusion vector corresponds to at least one segmentation mask result; The second fusion vector is processed by a fourth multilayer perceptron to obtain a second target fusion vector, wherein the dimension of the second target fusion vector is the same as the dimension of the first target fusion vector, and the second target fusion vector corresponds to at least one average intersection-and-union ratio, and the at least one average intersection-and-union ratio corresponds to the at least one segmentation mask result.

7. The method according to any one of claims 1 to 6, characterized in that After segmenting the image to be segmented based on the image segmentation result to obtain the target area in the image to be segmented, the method further includes: Obtaining decoding parameters corresponding to the decoder module and segmentation parameters corresponding to the segmentation header module, and obtaining a preset segmentation result; Inputting the image segmentation result and the preset segmentation result into a preset loss function to determine a corresponding loss value; Based on the loss value, the decoding parameters and the segmentation parameters are adjusted.

8. An image segmentation device, characterized in that: include: A receiving module, configured to receive an image to be segmented and text information describing the image to be segmented; A first processing module is configured to determine image features, image embedding features and text embedding features corresponding to the image to be segmented based on the image to be segmented and the text information; a second processing module configured to process the image features, the image embedding features and the text embedding features through a decoder module in a segmentation decoder based on cross attention and a multi-layer perceptron to obtain a first image vector and a first text vector, and obtain a first fusion vector based on the first image vector and the first text vector; a third processing module, configured to obtain an image segmentation result based on the first image vector, the first text vector and the first fusion vector through a segmentation head module in a segmentation decoder; The segmentation module is configured to segment the image to be segmented based on the image segmentation result to obtain a target area in the image to be segmented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images