Image segmentation method and device, electronic equipment and readable storage medium
Through the combination of text query enhancement module and target encoding and decoder, the problem of unsatisfactory image segmentation effect in complex scenarios is solved, and better global context understanding and segmentation effect is achieved.
Patent Information
- Application Number
- CN202411514152.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art cannot fully understand the global context information of the image in complex scenarios, resulting in unsatisfactory image segmentation effect.
By acquiring the input image and description text, the text query enhancement module is used to fusion and enhance feature fusion encoding and decoding, and the target decoding features are generated for image segmentation.
It enhances the model's understanding of cross-modal data, effectively captures global semantic information, overcomes the inefficiency problem of traditional full convolutional networks when dealing with long-distance dependence, and improves the image segmentation performance in complex scenarios.
Smart Images
Figure CN119540550A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to an image segmentation method, device, electronic device and readable storage medium. Background Art
[0002] The image segmentation task aims to segment the input image according to the natural language description. The existing technology usually uses a fully convolutional network (FCN) to complete image segmentation, but its method of implementing long-range dependency modeling by stacking small kernel convolutions is inefficient and not conducive to capturing the global context of the image. Secondly, some studies have attempted to apply the attention mechanism to the FCN-based segmentation model to establish the interaction between the input data and capture the global context information of the image. However, the existing method of introducing the attention mechanism is only used as an auxiliary in the FCN-based segmentation model, and cannot fully and effectively capture the global context information of the image in complex scenes. As a result, the image segmentation effect in complex scenes is not ideal.
[0003] It can be seen that the existing technology cannot fully understand the global context information in complex scenes, resulting in unsatisfactory image segmentation effects. Summary of the invention
[0004] In view of this, the embodiments of the present application provide an image segmentation method, device, electronic device and readable storage medium to solve the problem in the prior art that the global context information cannot be fully understood in complex scenes, resulting in unsatisfactory image segmentation effects.
[0005] In a first aspect of an embodiment of the present application, an image segmentation method is provided, the method comprising:
[0006] An input image and a description text are obtained, and the image features corresponding to the input image and the text features corresponding to the description text are determined; the image features and the text features are fused using a text query enhancement module, and the fused features are enhanced to obtain text query enhancement features; the image features and the text query enhancement features are fused and encoded using a target encoder to obtain target encoding features; the target encoding features and the text query enhancement features are fused and decoded using a target decoder to obtain target decoding features; the input image is segmented according to the target decoding features and the text query enhancement features to obtain a segmentation result of the input image.
[0007] According to a second aspect of the embodiments of the present application, there is provided an image segmentation device, the device comprising:
[0008] The acquisition module is configured to acquire an input image and a description text, and determine the image features corresponding to the input image and the text features corresponding to the description text; the enhancement module is configured to use the text query enhancement module to fuse the image features and the text features, and to enhance the fused features to obtain the text query enhancement features; the encoding module is configured to use the target encoder to fuse the image features and the text query enhancement features to obtain the target encoding features; the decoding module is configured to use the target decoder to fuse and decode the target encoding features and the text query enhancement features to obtain the target decoding features; the segmentation module is configured to segment the input image according to the target decoding features and the text query enhancement features to obtain the segmentation result of the input image.
[0009] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0010] According to a fourth aspect of an embodiment of the present application, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0011] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0012] The input image and description text are obtained, and the image features corresponding to the input image and the text features corresponding to the description text are determined; the image features and text features are fused using a text query enhancement module, and the fused features are enhanced to obtain text query enhancement features; the image features and text query enhancement features are fused and encoded using a target encoder to obtain target encoding features; the target encoding features and text query enhancement features are fused and decoded using a target decoder to obtain target decoding features; the input image is segmented according to the target decoding features and the text query enhancement features to obtain the segmentation result of the input image. This application enhances the model's ability to understand cross-modal data through the processing of input data by the text query enhancement module, so that the model can understand the description text from different angles. At the same time, the processing of the target encoder and the target decoder can enable the model to adaptively select the output characteristics of the text query enhancement module to adapt to different segmentation tasks and image content, thereby better generating masks; that is, this application enhances the overall understanding of multimodal features through global operations and effectively captures global semantic information. At the same time, the global processing approach overcomes the indirectness and inefficiency of traditional fully convolutional networks (FCNs) in dealing with long-distance dependencies, enabling the model to more accurately simulate the global context of visual and language information, thereby achieving better performance in image segmentation tasks in complex scenes and achieving better segmentation effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0014] Figure 1 It is a flowchart of an image segmentation method provided in an embodiment of the present application;
[0015] Figure 2 is a structural diagram of a text query enhancement module provided in an embodiment of the present application;
[0016] Figure 3 is a schematic diagram of the structure of a target encoder provided in an embodiment of the present application;
[0017] Figure 4 is a schematic diagram of the structure of a target decoder provided in an embodiment of the present application;
[0018] Figure 5 It is a flowchart of another image segmentation method provided in an embodiment of the present application;
[0019] Figure 6 is a structural schematic diagram of an image segmentation device provided in an embodiment of the present application;
[0020] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0022] An image segmentation method, device, electronic device, and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0023] Figure 1 FIG. 1 is a flow chart of an image segmentation method provided in an embodiment of the present application. Figure 1 As shown, the method includes:
[0024] S101, obtaining an input image and a description text, and determining image features corresponding to the input image and text features corresponding to the description text;
[0025] S102, using a text query enhancement module to perform feature fusion on the image features and the text features, and perform feature enhancement on the fused features to obtain text query enhancement features;
[0026] S103, using a target encoder to perform fusion encoding on the image features and the text query enhancement features to obtain a target encoding feature;
[0027] S104, using a target decoder to fuse and decode the target encoding feature and the text query enhancement feature to obtain a target decoding feature;
[0028] S105, segmenting the input image according to the target decoding feature and the text query enhancement feature to obtain a segmentation result of the input image.
[0029] Specifically, the input image can be a picture with a relatively complex scene, for example, a complex indoor photo: the content of the photo is: multiple adults talking (sitting on the sofa); a long wooden coffee table with books, magazines and a cup of coffee on it; two gray fabric sofas placed opposite each other and a large floor-to-ceiling window with light-colored curtains hanging on the window, the curtains are half-open. Next to the window is a floor lamp with a white lampshade and a black lamp pole. The wall is off-white and has several modern art paintings hanging on it; there is a light-colored carpet on the floor. At this time, the description text is "Please segment out all the windows and their curtains in the picture". The goal at this time is to segment all the windows and their curtains in the picture. The segmentation result can be that all windows and their curtains are segmented out of the picture separately, or all windows and their curtains will be highlighted or marked with a specific color for clear identification. Other elements (such as sofas, coffee tables, people, paintings, carpets, etc.) will remain as they are or be presented in darker tones to highlight the windows and curtains.
[0030] It can be understood that after obtaining the input image and the description text, the image features corresponding to the input image and the text features corresponding to the description text are determined, aiming to transform the input image and the description text into data that can be understood by the model. Specifically, the input image can be subjected to feature extraction through an image encoder to obtain image features, and the description text can be subjected to feature extraction through a text encoder to obtain text features. The image encoder and the text encoder can be selected by the user or the model according to task requirements. This example does not specifically limit the specific types of the image encoder and the text encoder.
[0031] In some instances, the obtained image features and text features will be input into the text query enhancement module for feature fusion and feature enhancement. The text query enhancement module in this example can combine text features and image features, and at the same time enhance the model's ability to understand cross-modal data, so that the model can understand the descriptive text from different angles, and finally obtain the text query enhancement features output by the text query enhancement module. It can be understood that the specific structure and workflow of the text query enhancement module will be explained in subsequent embodiments and will not be elaborated here.
[0032] In some examples, a target encoder is used to fuse and encode image features and text query enhancement features to obtain target encoding features. Exemplarily, the target encoder can be a transformer encoder. The following example uses the transformer encoder for illustrative purposes. The transformer encoder can capture the dependency between any two elements in a sequence and is very suitable for processing long-distance dependency problems. At the same time, the transformer encoder can simultaneously process data in two modalities, image features and text query enhancement features, to achieve effective fusion between the two modal data, so that the model can fully understand the input information and improve the accuracy and efficiency of task completion.
[0033] In some examples, the target decoder is used to fuse and decode the target encoding features and the text query enhancement features to obtain the target decoding features; exemplarily, the target decoder can be a transformer decoder. The following example uses the transformer decoder for illustrative purposes. The transformer decoder in this example can capture the dependency between any two positions in the input sequence. This ability is particularly important when processing sequence data such as natural language because it allows the model to understand and generate complex grammatical structures and semantic information. During the decoding process, the Transformer decoder not only considers its own input (i.e., the text query enhancement features), but also combines the output features of the target encoder (target encoding features). This mechanism enables the decoder to make full use of the target encoder's in-depth understanding of the input sequence to generate a more accurate and relevant output sequence (target decoding features) to adapt to different segmentation tasks and image content, thereby better generating masks.
[0034] It can be understood that the input image can be segmented according to the target decoding features and the text query enhancement features to obtain the segmentation result of the input image. The specific segmentation algorithm can be used to segment the input image, such as threshold-based segmentation, region growing, deep learning algorithm, etc. This example is not specifically limited here.
[0035] According to the technical solution provided in the embodiment of the present application, an input image and a description text are obtained, and the image features corresponding to the input image and the text features corresponding to the description text are determined; the image features and the text features are fused using a text query enhancement module, and the fused features are enhanced to obtain text query enhancement features; the image features and the text query enhancement features are fused and encoded using a target encoder to obtain target encoding features; the target encoding features and the text query enhancement features are fused and decoded using a target decoder to obtain target decoding features; the input image is segmented according to the target decoding features and the text query enhancement features to obtain the segmentation result of the input image. The present application enhances the model's ability to understand cross-modal data through the processing of input data by the text query enhancement module, so that the model can understand the description text from different angles. At the same time, the processing of the target encoder and the target decoder can enable the model to adaptively select the output characteristics of the text query enhancement module to adapt to different segmentation tasks and image content, thereby better generating masks; that is, the present application enhances the overall understanding of multimodal features through global operations and effectively captures global semantic information. At the same time, the global processing approach overcomes the indirectness and inefficiency of traditional fully convolutional networks (FCNs) in dealing with long-distance dependencies, enabling the model to more accurately simulate the global context of visual and language information, thereby achieving better performance in image segmentation tasks in complex scenes and achieving better segmentation effects.
[0036] In some embodiments, the text query enhancement module includes a first cross attention layer, an image-text cross-modal query processing model layer, a second cross attention layer, and a multi-layer perceptron layer. The text query enhancement module is used to perform feature fusion on image features and text features, and feature enhancement on the fused features to obtain text query enhancement features, including:
[0037] The image features are input into the image-text cross-modal query processing model layer for feature extraction to obtain the key features of the image; the text features and image features are fused using the first cross-attention layer to obtain the initial fused features; the image key features and the initial fused features are fused using the second cross-attention layer to obtain the target fused features; the target fused features are processed using the multi-layer perceptron layer, and the processed features are added to the text features to achieve feature enhancement to obtain text query enhanced features.
[0038] In order to better understand this embodiment, Figure 2 This embodiment is described as follows. Figure 2 As shown:
[0039] Specifically, the text query enhancement module includes a first cross-attention layer, an image-text cross-modal query processing model layer, a second cross-attention layer and a multi-layer perceptron layer. This example first uses the first cross-attention layer to fuse the text features and image features to obtain initial fused features. That is, in the fusion process of the first cross-attention layer, the text features are used as queries (Q), and the image features are used as keys (K) and values (V), so that the text features can be updated based on the image features, thereby capturing the key information in the image, and integrating this information into the text features to enhance the expressiveness of the text features.
[0040] It should be noted that the text features are used as queries (Q) and the image features are used as keys (K) and values (V), which means:
[0041] Query (Q): represents the information that you want to focus on or query, which refers to text features. In cross-modal tasks, text features contain the descriptive information that you want to segment from the image.
[0042] Key (K): An element used to match or compare with the query (Q), which refers to the image features. In the attention mechanism, the key (K) is usually used to calculate the similarity or relevance with the query (Q).
[0043] Values (V): Once the relevance between the query (Q) and the key (K) is determined, the values (V) are used to aggregate or transfer information. Here, the values (V) are also image features, but their role in the attention mechanism is to weight or filter them according to the relevance between the query (Q) and the key (K) so as to ultimately generate an optimized output that is relevant to the query (Q). In short, this setting allows the model to focus on the parts of the image that are relevant to the text features through the query (Q) and use this relevant information to perform the subsequent image segmentation task.
[0044] At the same time, the image features are extracted using the image text cross-modal query processing model layer (BLIP-Qformer). BLIP-Qformer can extract more advanced and richer semantic features from the image. These features are more abstract and representative than the original image features, providing a better foundation for subsequent feature fusion. In the attention mechanism, Q (Query), K (Key), and V (Value) are three key concepts. They are usually used to measure the correlation between different elements and aggregate or transfer information accordingly.
[0045] Furthermore, after the first cross-attention, the image and text features have been fused to a certain extent. However, in order to further deepen this fusion and improve the expressiveness of the features, this example uses the second cross-attention layer to fuse the key features of the image and the initial fused features to obtain the target fused features. This time, the initial fused features obtained by the first cross-attention are used as the new query (Q), and the key features of the image extracted by BLIP-Qformer are used as the key (K) and value (V), and the features are fused again through the cross-attention mechanism. This makes the interaction between the image and text features more sufficient and the fusion more in-depth.
[0046] Furthermore, the target fusion features are processed using a multi-layer perceptron layer. The multi-layer perceptron (MLP) has a powerful nonlinear transformation capability and can further integrate and refine the output of the secondary cross attention. Through the processing of MLP, redundant information can be removed, the expressive power of useful features can be enhanced, and the final features can be more compact and representative. Finally, the output of MLP is added to the text features to achieve feature enhancement. The purpose of this step is to effectively integrate image information into text features while retaining the information of text features. Through the addition operation, the originality of the text features is retained, and the contribution of image information is increased, making the final text features more comprehensive and descriptive.
[0047] In addition, in some embodiments, the target encoder includes a self-attention layer and a cross-attention layer, and the target encoder is used to perform fusion encoding on the image features and the text query enhancement features to obtain target encoding features, including:
[0048] The self-attention layer is used to process the image features to obtain the image self-attention features; the cross-attention layer is used to fuse the image self-attention features and the text query enhancement features to obtain the target encoding features.
[0049] In order to better understand this embodiment, Figure 3 This embodiment is described as follows. Figure 3 As shown:
[0050] Specifically, the target encoder includes a self-attention layer and a cross-attention layer. This example first uses the self-attention layer to process the image features to obtain the image self-attention features. The self-attention layer allows different parts of the image to "communicate" with each other, captures the contextual information inside the image, and generates image self-attention features. This process not only enhances the spatial coherence of the image features, but also improves the effectiveness of the image features in the subsequent fusion process.
[0051] Furthermore, the image self-attention feature is used as the query (Q), and the text query enhancement feature is used as the key (K) and value (V). The cross-attention layer is used to fuse the image self-attention feature and the text query enhancement feature to obtain the target encoding feature. In the cross-attention layer, the text query enhancement feature plays the role of a guide. The rich text information and key image features it carries guide the image self-attention feature to be more targeted. In this process, the semantic information in the text features is effectively mapped to the image features, achieving a deeper understanding and connection between images and texts.
[0052] In this way, through the collaborative work of the self-attention layer and the cross-attention layer, the target encoder not only strengthens the internal consistency of the image features, but also promotes the cross-modal fusion between the image and text features. The target encoding features finally obtained are a comprehensive reflection of the dual information of image and text, which not only retains the spatial structure and visual details of the image, but also integrates the semantic understanding and description ability of the text, providing strong support for subsequent task processing.
[0053] In addition, in some embodiments, the target encoder includes at least two sub-encoding modules, each of which includes a self-attention layer and a cross-attention layer, and before obtaining the target encoding feature, it also includes:
[0054] The output of the current sub-encoding module is used as the input of the next sub-encoding module, and the processing flow of the self-attention layer and the cross-attention layer in each sub-encoding module is repeatedly executed to iterate the features output by each sub-encoding module until the target fusion features output by the last sub-encoding module are obtained.
[0055] Specifically, in order to further improve the richness and accuracy of the target coding features, the target encoder in this embodiment adopts a multi-level sub-coding module structure. Each sub-coding module includes a self-attention layer and a cross-attention layer, and gradually deepens the degree of fusion between image and text features through layer-by-layer iteration. In the iterative process, the input image features are first processed using the self-attention layer of the current sub-coding module to generate image self-attention features. This process aims to enhance the autocorrelation of internal features of the image and capture more refined image context information. Subsequently, the image self-attention features are fused with the text query enhancement features using the cross-attention layer, and the deep interaction of image and text features is achieved through the cross-attention mechanism. At this point, the rich semantic information in the text query enhancement features will guide the image features to be adjusted more targeted, and promote the alignment and fusion of image and text features at the semantic level.
[0056] It can be understood that after completing the processing of the current sub-coding module, its output is used as the input of the next sub-coding module, and the processing flow of the self-attention layer and the cross-attention layer is repeated. As the number of iterations increases, the degree of fusion between the image and text features continues to deepen, and the feature expression ability is significantly improved. Finally, when the preset number of iterations is reached or specific conditions are met, the iteration process is stopped, and the output of the last sub-coding module is taken as the target fusion feature. This multi-level, iterative sub-coding module structure not only improves the richness and accuracy of the target coding features, but also enhances the flexibility and scalability of the model. By adjusting the number and parameter settings of the sub-coding modules, it can flexibly adapt to the needs and complexity of different tasks. At the same time, since each sub-coding module contains independent self-attention layers and cross-attention layers, data can be processed in parallel to improve the computational efficiency and response speed of the model.
[0057] According to the technical solution provided by the embodiment of the present application, by introducing a multi-level sub-coding module structure, the target encoder in this embodiment further improves the richness and accuracy of the target coding features while maintaining the deep fusion of image and text features. This provides a more comprehensive and accurate feature representation for subsequent task processing, which helps to improve the overall performance and effect of the model.
[0058] In some embodiments, the target decoder includes a self-attention layer and a cross-attention layer, and the target decoder is used to fuse and decode the target encoding features and the text query enhancement features to obtain target decoding features, including:
[0059] The text query enhancement features are processed using the self-attention layer to obtain the processed text query enhancement features; the target fusion features and the processed text query enhancement features are fused using the cross-attention layer to obtain the target decoding features.
[0060] In order to better understand this embodiment, Figure 4 This embodiment is described as follows. Figure 4 As shown:
[0061] Specifically, in the design of the target decoder, this example also uses a combination of self-attention layer and cross-attention layer to achieve efficient fusion decoding of target encoding features and text query enhancement features. First, the text query enhancement features are processed using the self-attention layer. This step aims to strengthen the semantic associations within the text query so that the text query can more accurately capture the user's intentions and query focus. Through the self-attention mechanism, each word in the text query can establish connections with other words, thereby generating more accurate and context-rich processed text query enhancement features.
[0062] Furthermore, the processed text query enhancement features are used as query (Q), and the output of the target encoder is used as key (K) and value (V). The cross-attention layer is used to fuse and decode the processed text query enhancement features with the target encoding features. In the cross-attention layer, the target encoding features, as a comprehensive reflection of the dual information of image and text, provide rich visual and semantic context for text queries. The processed text query enhancement features guide the target encoding features to be decoded in a targeted manner with their precise semantic expression and query intent. Through the cross-attention mechanism, the key information in the text query can be accurately mapped to the corresponding part of the target encoding features, realizing the deep fusion and complementarity of image and text features. Finally, after the fusion decoding of the cross-attention layer, the target decoding features are obtained. This feature not only retains the precise semantics of the text query, but also incorporates the spatial structure and visual details of the image, forming a comprehensive representation of the dual understanding of the query intent and image content. Such target decoding features provide strong support for subsequent task processing.
[0063] According to the technical solution provided in the embodiment of the present application, the text query enhancement feature is processed using the self-attention layer to obtain the processed text query enhancement feature; the target fusion feature and the processed text query enhancement feature are fused using the cross-attention layer to obtain the target decoding feature. The obtained target decoding feature not only retains the precise semantics of the text query, but also incorporates the spatial structure and visual details of the image, forming a comprehensive representation of the dual understanding of the query intent and the image content.
[0064] In addition, in some embodiments, the target decoder includes at least two sub-decoding modules, each sub-decoding module includes a self-attention layer and a cross-attention layer, and before obtaining the target decoding feature, it also includes:
[0065] The output of the current sub-decoding module is used as the input of the next sub-decoding module, and the processing flow of the self-attention layer and the cross-attention layer in each sub-decoding module is repeatedly executed to iterate the features output by each sub-decoding module until the target decoding features output by the last sub-decoding module are obtained.
[0066] Specifically, in order to further enhance the richness and depth of the target decoding features, the target decoder in this embodiment adopts a multi-level sub-decoding module structure. This structure allows for more detailed and in-depth fusion decoding of the target encoding features and the text query enhancement features. Each sub-decoding module independently includes a self-attention layer and a cross-attention layer, and gradually optimizes and enriches the feature representation in the decoding process through a layer-by-layer iteration. In the iterative process, the text query enhancement features are first processed using the self-attention layer of the current sub-decoding module. This step not only strengthens the semantic association within the text query, but also enables the text query to capture the user's intention and query focus more accurately. Through the self-attention mechanism, each word in the text query can establish a connection with other words to generate a more accurate and context-rich processed text query enhancement feature.
[0067] Furthermore, the processed text query enhanced features are used as query (Q), and the output of the target encoder (i.e., target fusion features) are used as key (K) and value (V), and are sent to the cross-attention layer of the current sub-decoding module for fusion decoding. In the cross-attention layer, the target fusion features, as a comprehensive reflection of the dual information of image and text, provide rich visual and semantic context for text queries. The processed text query enhanced features, with their precise semantic expression and query intent, guide the target fusion features for targeted decoding. Through the cross-attention mechanism, the key information in the text query can be accurately mapped to the corresponding part of the target fusion features, realizing the deep fusion and complementarity of image and text features.
[0068] It can be understood that after completing the processing of the current sub-decoding module, its output is used as the input of the next sub-decoding module, and the processing flow of the self-attention layer and the cross-attention layer is repeated. As the number of iterations increases, the feature representation in the decoding process is continuously optimized and enriched, and the expressive power of the target decoding feature is significantly improved. Finally, when the preset number of iterations is reached or specific conditions are met, the iteration process is stopped, and the output of the last sub-decoding module is taken as the final target decoding feature. This multi-level, iterative sub-decoding module structure not only improves the richness and accuracy of the target decoding features, but also enhances the flexibility and scalability of the model. By adjusting the number and parameter settings of the sub-decoding modules, it can flexibly adapt to the needs and complexity of different tasks. At the same time, since each sub-decoding module contains independent self-attention layers and cross-attention layers, data can be processed in parallel, further improving the computational efficiency and response speed of the model.
[0069] In this way, the target decoder in this embodiment achieves efficient fusion decoding of target encoding features and text query enhancement features by introducing a multi-level sub-decoding module structure. This structure not only improves the richness and accuracy of target decoding features, but also provides a more comprehensive and accurate feature representation for subsequent task processing, which helps to improve the overall performance and effect of the model.
[0070] In addition, in some embodiments, the input image is segmented according to the target decoding feature and the text query enhancement feature to obtain a segmentation result of the input image, including:
[0071] The text query enhancement features are processed using a multi-layer perceptron layer, and the processed text query enhancement features are concatenated with the target decoding features to obtain target concatenated features; the target concatenated features are processed using a linear layer to obtain target linear features, and confidence calculation is performed on the target linear features to obtain confidence calculation results; the confidence calculation results are multiplied with the target decoding features, and image segmentation is performed based on the multiplication result to obtain segmentation results.
[0072] Specifically, the input image is segmented according to the target decoding features and the text query enhancement features to obtain the segmentation results of the input image. First, the text query enhancement features are nonlinearly transformed using the multi-layer perceptron (MLP) layer to extract their deeper semantic representations. In order to strengthen the key information in the text query and make it complementary to the target decoding features at a higher level. Subsequently, the processed text query enhancement features are spliced with the target decoding features to generate target spliced features. This splicing operation combines the dual information from the text and the image. The target decoding features provide context and global information, while the processing results of the multi-layer perceptron (MLP) layer focus more on detailed semantic features, ultimately forming a feature representation of the comprehensive understanding of the query intent and image content.
[0073] Furthermore, the linear layer is used to perform a linear transformation on the target splicing features to generate target linear features. The purpose is to map the spliced feature space to a feature space that is more suitable for image segmentation, that is, to convert the spliced results into category scores for each pixel, which provides a basis for subsequent confidence calculations and helps to simplify complex feature representations into category scores that are easy to understand and process. Subsequently, the confidence of the target linear features is calculated, and the output of the linear layer is converted into confidence probabilities, so that the model can assign a numerical value to each pixel indicating the possibility of it belonging to each category. It helps to better weigh the possibilities of different categories during the segmentation process and reduce misclassification, that is, to evaluate the contribution of each feature point to the image segmentation results. The confidence calculation results reflect the reliability of feature points in different segmentation areas, providing an important basis for subsequent segmentation decisions.
[0074] Further, the confidence calculation result and the target decoded feature are multiplied, and the image is segmented according to the multiplication result to obtain the segmentation result. By multiplying the confidence calculation result with the output of the Transformer decoder, the model is able to combine global context information and local semantic features to assign the final category to each pixel in the image. A variety of factors are considered comprehensively to generate more accurate and detailed segmentation results. Secondly, by comprehensively considering the confidence calculation results and the output of the Transformer decoder, the model can more accurately identify different areas in the image and reduce mis-segmentation and missed segmentation. It helps to improve the accuracy and reliability of the segmentation results. In addition, because this example combines global and local information, the model is better able to adapt to complex image scenes, including those with challenges such as complex textures, lighting changes, and occlusion. This makes the model more generalizable and practical in practical applications.
[0075] According to the technical solution provided in the embodiment of the present application, the text query enhancement feature is processed using a multi-layer perceptron layer, and the processed text query enhancement feature is spliced with the target decoding feature to obtain the target splicing feature; the target splicing feature is processed using a linear layer to obtain the target linear feature, and the confidence calculation is performed on the target linear feature to obtain the confidence calculation result; the confidence calculation result is multiplied with the target decoding feature, and the image is segmented according to the multiplication result to obtain the segmentation result. Not only the rich information in the target decoding feature and the text query enhancement feature is fully utilized, but also the features are further optimized and recalibrated by means of multi-layer perceptron, linear layer and confidence calculation. Not only the accuracy and precision of image segmentation are improved, but also the model's ability to understand and respond to complex query intent is enhanced.
[0076] Figure 5 FIG. 1 is a flow chart of another image segmentation method provided in an embodiment of the present application. Figure 5 As shown, the method includes:
[0077] First, the input image and description text are obtained, and the image features corresponding to the input image and the text features corresponding to the description text are determined by the image encoder and the text encoder respectively, and the image features and text features are fused by the text query enhancement module, and the fused features are enhanced to obtain the text query enhanced features;
[0078] Then, the target encoder is used to fuse the image features and the text query enhancement features to obtain the target encoding features, and the target decoder is used to fuse the target encoding features and the text query enhancement features to obtain the target decoding features.
[0079] Finally, the text query enhancement features are processed using a multi-layer perceptron layer, and the processed text query enhancement features are concatenated with the target decoding features to obtain the target concatenated features; the target concatenated features are processed using a linear layer to obtain the target linear features, and the confidence of the target linear features is calculated to obtain the confidence calculation results; the confidence calculation results are multiplied with the target decoding features, and image segmentation is performed based on the multiplication result to obtain the segmentation results.
[0080] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.
[0081] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.
[0082] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0083] Figure 6 Schematic diagram of the structure of an image segmentation device provided in an embodiment of the present application. Figure 6 As shown, the device comprises:
[0084] The acquisition module 601 is configured to acquire an input image and a description text, and determine image features corresponding to the input image and text features corresponding to the description text;
[0085] The enhancement module 602 is configured to use the text query enhancement module to perform feature fusion on the image feature and the text feature, and perform feature enhancement on the fused feature to obtain a text query enhanced feature;
[0086] The encoding module 603 is configured to use a target encoder to perform fusion encoding on the image feature and the text query enhancement feature to obtain a target encoding feature;
[0087] The decoding module 604 is configured to use a target decoder to fuse and decode the target encoding feature and the text query enhancement feature to obtain a target decoding feature;
[0088] The segmentation module 605 is configured to segment the input image according to the target decoding feature and the text query enhancement feature to obtain a segmentation result of the input image.
[0089] In some embodiments, the text query enhancement module includes a first cross-attention layer, an image-text cross-modal query processing model layer, a second cross-attention layer and a multi-layer perceptron layer. The enhancement module 602 is also configured to input image features into the image-text cross-modal query processing model layer for feature extraction to obtain image key features; use the first cross-attention layer to fuse text features and image features to obtain initial fused features; use the second cross-attention layer to fuse image key features and initial fused features to obtain target fused features; use the multi-layer perceptron layer to process the target fused features, and add the processed features to the text features to achieve feature enhancement to obtain text query enhanced features.
[0090] In some embodiments, the target encoder includes a self-attention layer and a cross-attention layer, and the encoding module 603 is also configured to use the self-attention layer to process the image features to obtain image self-attention features; and use the cross-attention layer to fuse the image self-attention features and the text query enhancement features to obtain target encoding features.
[0091] In some embodiments, the target encoder includes at least two sub-encoding modules, each sub-encoding module includes a self-attention layer and a cross-attention layer, and the encoding module 603 is also configured to use the output of the current sub-encoding module as the input of the next sub-encoding module, and repeatedly execute the processing flow of the self-attention layer and the cross-attention layer in each sub-encoding module so as to iterate the features output by each sub-encoding module until the target fusion features output by the last sub-encoding module are obtained.
[0092] In some embodiments, the target decoder includes a self-attention layer and a cross-attention layer, and the decoding module 604 is also configured to use the self-attention layer to process the text query enhancement feature to obtain the processed text query enhancement feature; and use the cross-attention layer to fuse the target fusion feature and the processed text query enhancement feature to obtain the target decoding feature.
[0093] In some embodiments, the target decoder includes at least two sub-decoding modules, each sub-decoding module includes a self-attention layer and a cross-attention layer, and the decoding module 604 is also configured to use the output of the current sub-decoding module as the input of the next sub-decoding module, and repeatedly execute the processing flow of the self-attention layer and the cross-attention layer in each sub-decoding module so as to iterate the features output by each sub-decoding module until the target decoding features output by the last sub-decoding module are obtained.
[0094] In some embodiments, the segmentation module 605 is also configured to process the text query enhancement features using a multi-layer perceptron layer, and concatenate the processed text query enhancement features with the target decoding features to obtain target concatenated features; process the target concatenated features using a linear layer to obtain target linear features, and perform confidence calculation on the target linear features to obtain confidence calculation results; multiply the confidence calculation results and the target decoding features, and perform image segmentation based on the multiplication result to obtain a segmentation result.
[0095] Figure 7 Schematic diagram of an electronic device 7 provided in an embodiment of the present application. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0096] The electronic device 7 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 7 may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 The electronic device 7 is merely an example and does not limit the electronic device 7 , and may include more or less components than those shown in the figure, or different components.
[0097] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0098] The memory 702 may be an internal storage unit of the electronic device 7, for example, a hard disk or memory of the electronic device 7. The memory 702 may also be an external storage device of the electronic device 7, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. The memory 702 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 702 is used to store computer programs and other programs and data required by the electronic device.
[0099] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0100] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0101] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. An image segmentation method, characterized in that: include: Acquire an input image and a description text, and determine image features corresponding to the input image and text features corresponding to the description text; Using a text query enhancement module to perform feature fusion on the image features and the text features, and perform feature enhancement on the fused features to obtain text query enhancement features; Using a target encoder to perform fusion encoding on the image feature and the text query enhancement feature to obtain a target encoding feature; Using a target decoder to fuse and decode the target encoding feature and the text query enhancement feature to obtain a target decoding feature; The input image is segmented according to the target decoding feature and the text query enhancement feature to obtain a segmentation result of the input image.
2. The method according to claim 1, characterized in that The text query enhancement module includes a first cross attention layer, an image-text cross-modal query processing model layer, a second cross attention layer and a multi-layer perceptron layer. The text query enhancement module is used to perform feature fusion on the image features and the text features, and feature enhancement on the fused features to obtain text query enhancement features, including: Inputting the image features into the image-text cross-modal query processing model layer for feature extraction to obtain key image features; Using the first cross attention layer to fuse the text features and the image features to obtain an initial fused feature; Using the second cross attention layer to fuse the key image features with the initial fusion features to obtain a target fusion feature; The target fusion feature is processed by using the multi-layer perceptron layer, and the processed feature is added to the text feature to achieve feature enhancement, thereby obtaining the text query enhanced feature.
3. The method according to claim 1, characterized in that The target encoder includes a self-attention layer and a cross-attention layer. The target encoder is used to perform fusion encoding on the image feature and the text query enhancement feature to obtain a target encoding feature, including: Processing the image features using the self-attention layer to obtain image self-attention features; The image self-attention feature and the text query enhancement feature are fused using the cross attention layer to obtain the target encoding feature.
4. The method according to claim 3, characterized in that The target encoder includes at least two sub-encoding modules, each of which includes a self-attention layer and a cross-attention layer. Before obtaining the target encoding feature, the target encoder further includes: The output of the current sub-encoding module is used as the input of the next sub-encoding module, and the processing flow of the self-attention layer and the cross-attention layer in each sub-encoding module is repeatedly executed to iterate the features output by each sub-encoding module until the target fusion features output by the last sub-encoding module are obtained.
5. The method according to claim 1, characterized in that The target decoder includes a self-attention layer and a cross-attention layer. The target decoder is used to fuse and decode the target encoding feature and the text query enhancement feature to obtain a target decoding feature, including: Processing the text query enhancement feature using the self-attention layer to obtain a processed text query enhancement feature; The target fusion feature and the processed text query enhancement feature are fused using the cross attention layer to obtain the target decoding feature.
6. The method according to claim 5, characterized in that The target decoder includes at least two sub-decoding modules, each of which includes a self-attention layer and a cross-attention layer. Before obtaining the target decoding feature, the target decoder further includes: The output of the current sub-decoding module is used as the input of the next sub-decoding module, and the processing flow of the self-attention layer and the cross-attention layer in each sub-decoding module is repeatedly executed to iterate the features output by each sub-decoding module until the target decoding features output by the last sub-decoding module are obtained.
7. The method according to claim 1, characterized in that The step of segmenting the input image according to the target decoding feature and the text query enhancement feature to obtain a segmentation result of the input image includes: The text query enhancement feature is processed by using a multi-layer perceptron layer, and the processed text query enhancement feature is spliced with the target decoding feature to obtain a target splicing feature; Using a linear layer to process the target splicing feature to obtain a target linear feature, and performing confidence calculation on the target linear feature to obtain a confidence calculation result; The confidence calculation result and the target decoding feature are multiplied, and image segmentation is performed according to the multiplication result to obtain the segmentation result.
8. An image segmentation device, characterized in that: include: An acquisition module is configured to acquire an input image and a description text, and determine image features corresponding to the input image and text features corresponding to the description text; an enhancement module configured to perform feature fusion on the image feature and the text feature using the text query enhancement module, and perform feature enhancement on the fused feature to obtain a text query enhancement feature; An encoding module is configured to use a target encoder to perform fusion encoding on the image feature and the text query enhancement feature to obtain a target encoding feature; A decoding module is configured to use a target decoder to fuse and decode the target encoding feature and the text query enhancement feature to obtain a target decoding feature; The segmentation module is configured to segment the input image according to the target decoding feature and the text query enhancement feature to obtain a segmentation result of the input image.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.