Reference image segmentation method and system based on multi-level feature fusion

By employing a multi-level feature fusion method, text and image features are fused in a lightweight manner during reference image segmentation, thus solving the problem of hardware resource burden and achieving efficient image segmentation results.

CN119049058AActive Publication Date: 2024-11-29GUANGDONG UNIV OF PETROCHEMICAL TECH

Patent Information

Application Number
CN202411248576.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-29
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Existing multi-scale feature fusion methods have high computational overhead and heavy hardware resource burden in reference image segmentation, and the model complexity is also high.

Method used

A multi-level feature fusion method is adopted to encode text information into text features and the original image into multiple image feature maps of different scales. The image features and text features are fused through a lightweight fusion block. The mask features and text features are used for query decoding to generate a segmentation map of the target information.

Benefits of technology

The number of parameters in the fusion and decoding stages is reduced, which lowers the burden on hardware resources, while maintaining efficient fusion and segmentation accuracy of bimodal information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049058A_ABST
    Figure CN119049058A_ABST
Patent Text Reader

Abstract

The invention relates to the field of image processing, and relates to a reference image segmentation method and system based on multi-level feature fusion. The method comprises the following steps: encoding text information into text features, and encoding an original image with target information into a plurality of image feature maps with different scales; fusing the image feature maps of the plurality of scales with the text features respectively to generate a plurality of image feature maps of different scales fused with the text information; carrying out pixel decoding on the image feature map fused with the text information to obtain mask features and a plurality of enhanced image feature maps with different scales; and performing Query decoding on the enhanced image feature maps of different scales to obtain a segmentation map containing target information. By adopting the fusion strategy based on the lightweight fusion block, the parameter quantity in the fusion stage and the decoding stage is reduced, and the burden on hardware resources is further reduced. In addition, the fusion block is based on a single attention mechanism, the parameter quantity is small, the operation is simple, and the bimodal fusion level is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a reference image segmentation method and system based on multi-level feature fusion. Background Art

[0002] Reference image segmentation is a cross-modal task that involves processing two sensory information, visual and text. It processes a given image and a reference text pointing to an object in the image, locates and segments the target object under the guidance of the reference text, and generates a segmentation map of the area where the target object is located. This process involves pixel-level reasoning of both image and text modalities. To achieve a more ideal effect, it is necessary to fully align and deeply integrate image and text information.

[0003] Existing multi-scale feature fusion methods often use the attention mechanism, but the attention mechanism itself has a certain computational overhead, and the model is highly complex in the feature fusion stage, which places a heavy burden on hardware resources. Summary of the invention

[0004] In order to solve at least one of the above-mentioned defects, the present invention application proposes a reference image segmentation method and system based on multi-level feature fusion, which is used to solve the problem that multi-scale feature fusion places a heavy burden on hardware resources.

[0005] According to a first aspect of the present application, a reference image segmentation method based on multi-level feature fusion is provided, comprising:

[0006] Encode text information into text features, and encode the original image with target information into multiple image feature maps of different scales;

[0007] The image feature map of each scale is fused with the text features respectively to generate multiple image feature maps of different scales that incorporate text information;

[0008] The image feature maps of multiple scales that incorporate text information are decoded pixel by pixel to obtain mask features and enhanced image feature maps of multiple scales;

[0009] The mask features and the text features are used to perform query decoding on the enhanced image feature maps of the multiple different scales to obtain a segmentation map containing target information.

[0010] Preferably, during the fusion process, the image features of the multiple different scales are respectively subjected to an attention mechanism operation with the text features, and the obtained results are element-wise multiplied with the image feature projection results, and the result after the element-wise multiplication is projected to obtain image feature maps of multiple scales that fuse text information;

[0011] The one-time attention mechanism operation includes performing matrix multiplication on multiple image feature maps of different scales and the text feature, normalizing the obtained result, and performing matrix multiplication on the normalized result and the text feature.

[0012] Preferably, the fusion process is specifically expressed by the following formula:

[0013] Q q =V i W Q1 ,K k =FW K1 ,V v =FW V1

[0014]

[0015] Among them, the subscript i represents the i-th scale, V i represents the image feature map of the i-th scale, F represents the text feature, and W Q1 , W Q2 , W K1 , W V1 , W F are all linear transformation matrices; Q q Represents the image feature map of the i-th scale and W Q1 The result after multiplication; K k Represents text features and W K1 The result after multiplication; V v Represents text features and W V1 The result after multiplication; ⊙ represents element multiplication; Softmax represents the normalization function; d k Indicates the dimension of the key vector, Fuse i (V i ,F) represents the image feature map of the i-th scale that integrates text information, and n represents the total number of scales.

[0016] Based on the fusion process, each stage of image feature encoding is independent of the fusion operation and is not restricted by whether the fusion of the previous scale features is completed or not, thereby ensuring the flexibility of feature encoding.

[0017] Preferably, encoding the text information into text features specifically includes:

[0018] Preset text information, and convert the text information into a feature sequence;

[0019] Encoding the feature sequence to obtain text features,

[0020] and / or,

[0021] The encoding of the original image with target information into multiple image feature maps of different scales specifically includes:

[0022] The original image with target information is encoded, and a feature map is extracted once in each encoding stage to obtain multiple image feature maps of different scales.

[0023] In an optional solution, a word segmenter of a text encoder is used to convert the text information into a feature sequence, and specifically Bert-base can be selected as the text encoder;

[0024] The original image with target information is encoded using an image encoder. The image encoder can select different mainstream image encoders as the visual backbone network, such as Resnet50, Swin-Transformer-Tiny and

[0025] Swin-Transformer-Base, etc.

[0026] Preferably, the pixel decoding adopts a multi-scale deformable attention module and a feedforward neural network, and decodes the image feature maps of multiple scales that are integrated with text information based on the attention module and the feedforward neural network to interact between feature maps of different scales to obtain mask features and enhanced image feature maps of multiple scales.

[0027] Preferably, the Query decoding adopts a learnable Object Query decoder, the Object Query decoder includes multiple decoding layers, each decoding layer includes a text cross attention module, an image cross attention module and a feedforward neural network FFN, and the Query decoding process includes:

[0028] Assume that there are N decoding layers, initialize Query to represent the target information, and enter the first decoding layer as output;

[0029] For the 1st to Nth decoding layers, the output of the previous decoding layer is received and interacted with the text feature through a text cross attention module to generate a query carrying text information, the query carrying text information and the mask feature are processed by point multiplication and segmentation to obtain an attention mask, the attention mask interacts with the enhanced image feature map and the query carrying text information through an image cross attention module, and the result after the interaction is enhanced with a nonlinear expression capability through a feedforward neural network FFN as output;

[0030] For the Nth decoding layer, the result after interaction through the image cross attention module is enhanced with the nonlinear expression ability through the feedforward neural network FFN, and then processed with the mask feature through dot multiplication segmentation to obtain a segmentation map containing target information.

[0031] In an optional solution, in order to avoid the unstable Query-Target bipartite matching problem that may be caused by multiple Object Queries, the Query decoder adopts a single learnable Object Query, which can interact with the image and text information alternately, and update itself by alternately querying multi-scale image information and text information through multiple layers of Query decoder layers.

[0032] Before entering the image cross-attention module, the query carrying text information is first subjected to a point multiplication segmentation process with the mask feature to obtain a phased output, and the phased output is subjected to downsampling and binarization operations to obtain a binarization operation result, and a weighted masking operation is performed on the binarization operation result. The weighted masked result is used as the attention mask of the image cross-attention mechanism, and the attention mask softly acts as an area of ​​interest, excluding areas of no interest to reduce noise interference;

[0033] The Query carrying text information interacts with the enhanced image feature map and the attention mask through an image cross-attention module, and the result of the interaction enters the next decoding layer as output, and / or the result of the interaction is processed by dot multiplication segmentation with the mask feature to obtain a segmentation map containing target information.

[0034] Optionally, the Query decoding uses the weighted sum of the dice loss function and the cross entropy loss function ce as the final loss function, and the calculation formula of the final loss function is expressed as:

[0035]

[0036] Where N is the number of layers of the Query decoder, pred j is the stage output, j represents the jth decoding layer, target is the segmentation map containing target information, λ dice is the weight of the dice loss function, λ ce is the weight of the cross entropy loss function ce, dice_loss and ce_loss are the weighted results of the dice loss function and the cross entropy loss function ce respectively, and loss is the final loss function.

[0037] According to a second aspect of the present application, there is provided a reference image segmentation system based on multi-level feature fusion, comprising an encoding module, a pixel decoding module, a query decoding module and a fusion module;

[0038] The encoding module includes a text encoding module and an image encoding module, the text encoding module is used to encode text information into text features, and the image encoding module is used to encode the original image with target information into multiple image feature maps of different scales;

[0039] The fusion module is used to fuse the image feature map of each scale with the text feature respectively, to generate a plurality of image feature maps of different scales fused with text information;

[0040] The pixel decoding module is used to perform pixel decoding on the image feature maps of multiple scales that incorporate text information, to obtain mask features and enhanced image feature maps of multiple scales;

[0041] The query decoding module is used to perform query decoding on the multiple enhanced image feature maps of different scales using the mask features and the text features to obtain a segmentation map containing target information.

[0042] According to a third aspect of the present application, an electronic device is provided, including:

[0043] a memory for storing one or more computer programs;

[0044] The processor implements the reference image segmentation method based on multi-level feature fusion according to the first aspect when the one or more computer programs are executed by the processor.

[0045] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a reference image segmentation method based on multi-level feature fusion according to the first aspect when executed.

[0046] Based on any of the above aspects, the reference image segmentation method and system based on multi-level feature fusion provided by the embodiment of the present application adopts a fusion strategy based on a lightweight fusion block to reduce the number of parameters in the fusion image and text feature stage and the decoding stage, thereby reducing the burden on hardware resources. In addition, the fusion block is based on a single attention mechanism, which ensures the fusion level of dual modalities while having a small number of parameters and simple operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of a reference image segmentation method based on multi-level feature fusion provided in this embodiment.

[0049] Figure 2 An overall framework diagram of a reference image segmentation method based on multi-level feature fusion provided in this embodiment.

[0050] Figure 3 A schematic diagram of the fusion process of a reference image segmentation method based on multi-level feature fusion provided in this embodiment.

[0051] Figure 4 A visualization of a feature attention map of a reference image segmentation method based on multi-level feature fusion provided in this embodiment.

[0052] Figure 5 This is a main structural diagram of a pixel decoder and a query decoder of a reference image segmentation method based on multi-level feature fusion provided in this embodiment.

[0053] Figure 6 A schematic diagram of the functional modules of a reference image segmentation system based on multi-level feature fusion provided in this embodiment.

[0054] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this embodiment.

[0055] Figure 8 A schematic application scenario diagram of a reference image segmentation method based on multi-level feature fusion provided in this embodiment.

[0056] Fig. 9 The system, fusion block and pixel decoder parameter data table provided for this embodiment when using different image encoders. Icon: 100, server; 200, terminal; 501, text cross attention module Attn t ; 502, image cross attention module Attn v ; 503, feed-forward neural network FFN; 504, Object Query; 601, encoding module; 602, fusion module; 603, pixel decoding module; 604, Query decoding module; 710, electronic device; 711, memory; 712, processor; 713, communication module; 714, input / output interface; 715, bus. DETAILED DESCRIPTION

[0057] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] Existing multi-scale feature fusion methods often use the attention mechanism, but the attention mechanism itself has a certain computational overhead, and the model is highly complex in the feature fusion stage, which places a heavy burden on hardware resources.

[0061] On the one hand, the embodiment of the present application provides a reference image segmentation method based on multi-level feature fusion, such as Figure 1 As shown, the method at least includes:

[0062] S1, encode the text information into text features, and encode the original image with target information into multiple image feature maps of different scales;

[0063] S2, fusing the image feature map of each scale with the text feature respectively, to generate multiple image feature maps of different scales that fuse text information;

[0064] S3, performing pixel decoding on the image feature maps of multiple scales that incorporate text information to obtain mask features and enhanced image feature maps of multiple scales;

[0065] S4. Query decoding is performed on the enhanced image feature maps of the multiple different scales using the mask feature and the text feature to obtain a segmentation map containing target information.

[0066] The following is a specific example and Figure 2 , the technical solution provided in this application is described in detail.

[0067] S1. Encode text information into text features and encode the original image with target information into multiple image feature maps of different scales.

[0068] In this embodiment, the text encoder G is used t Encode the text information into text features F and use the image encoder G v , encode the original image with target information into multiple image feature maps V of different scales, where p represents the total number of scales.

[0069] Optionally, Bert-base is selected as the text encoder G in the embodiment of the present application. t In this embodiment, the text encoding process may include:

[0070] Preset text information T, use text encoder G t The word segmenter converts it into a token sequence e, and then the token sequence e is sent to the text encoder G t Get the text feature F, The encoding process can be expressed as:

[0071]

[0072] Among them, L is the sequence length, C l Represents the number of text feature dimensions, The parameter quantity representing the text features.

[0073] Optionally, the image encoder G v Different mainstream image encoders can be used as visual backbone networks, such as Res-50, Swin-Tiny, and Swin-Base.

[0074] Exemplarily, in this embodiment, the process of image encoding includes:

[0075] For a three-channel image with height and width H and W respectively Using the image encoder G v The process of extracting image feature maps at four scales can be expressed as:

[0076]

[0077] in, Among them, H i and W i Represent the height and width of the i-th scale image feature map, respectively. Represents the image feature dimension of the i-th image channel, i represents the i-th scale, the height and width of the four scale feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image height and width respectively. Each scale feature map is extracted from an encoding stage. i represents the image feature map of the i-th scale, V represents the feature map set, I i Represents the image feature map of the i-th size.

[0078] S2. Fuse the image feature map of each scale with the text features respectively to generate multiple image feature maps of different scales that fuse text information.

[0079] In this embodiment, a lightweight fusion block based on a single attention mechanism is used to merge the multiple image feature maps V at different scales. i The attention mechanism operation is performed on the text feature F respectively, where the image feature acts as the query and the text feature acts as the key and value. The specific process includes:

[0080] The multiple image feature maps V of different scales are i The results are normalized by performing matrix multiplication with the text feature F, and the results are respectively normalized by performing matrix multiplication with the text feature F, and the results are respectively normalized with the corresponding image feature map V i The projection result is element-wise multiplied, and the result after element-wise multiplication is projected to obtain multiple image feature maps V of different scales that integrate text information. i ′ ,

[0081] The main function of fusion is to integrate image features of different semantic levels and scales into text information, so that image and text features of different levels can perceive text information and align image semantics with text semantics.

[0082] Specifically, Figure 3 As shown, the fusion process can be specifically expressed by the following formula:

[0083] Q q =V i W Q1 ,K k =FW K1 ,V v =FW V1

[0084]

[0085] Among them, the subscript i represents the i-th scale, V′ represents the image feature map integrated with text features, and V i represents the image feature map of the i-th scale, F represents the text feature, and W Q1 , W Q2 , W K1 , W V1 , W F are all linear transformation matrices; Q q Represents the image feature map of the i-th scale and W Q1 The result after multiplication; K k Represents text features and W K1 The result after multiplication; V v Represents text features F and W V1 The result after multiplication; ⊙ represents element multiplication; Softmax represents the normalization function; d k Indicates the dimension of the key vector, Fuse i (V i ,F) represents the image feature map of the i-th scale that integrates text information.

[0086] For example, Figure 4 Represents the visualization of the feature attention map within the fusion block during the fusion process. The original image is above the text, (a) to (d) are the attention maps of the input image features and the word “wheelchair” in the input text, and (e) is the final mask of the model for the target image.

[0087] It can be understood that (a) shows that in the initial stage of image and text fusion, the fusion block focuses on the lower semantic level of image features, mainly focusing on image texture, contour and key points, which is similar to the properties of traditional convolutional neural networks. The evolution of the attention map from (a) to (d) shows that as the image encoding stage proceeds, the degree of fusion of image and text features gradually deepens, and the focus area of ​​the fusion block gradually changes from the global texture contour information to the location information of the local reference object referred to by the text content.

[0088] Based on the fusion process, each stage of image feature encoding is independent of the fusion operation and is not restricted by whether the fusion of the previous scale features is completed or not, thereby ensuring the flexibility of feature encoding.

[0089] S3. The image feature maps of multiple scales that incorporate text information are pixel decoded to obtain mask features and enhanced image feature maps of multiple scales.

[0090] In an optional implementation, a Multi-Scale Deformable with a multi-scale deformable attention mechanism can be used as the pixel decoder G p , used to model the interaction between feature maps of different scales, the pixel decoder Gp It comprises multiple pixel decoding layers, each of which comprises a multi-scale deformable attention module and a feedforward neural network with two layers of nonlinear activation functions. The pixel decoder G p Decode the image feature maps of multiple scales that incorporate text information to obtain mask features and multiple enhanced image feature maps of different scales. The pixel decoder G p The decoding process can be expressed as:

[0091]

[0092] Among them, Fv i represents the enhanced image feature map of the i-th scale, M is the mask feature, V i ′ is the i-th scale image feature map that integrates text features. represents the parameter quantity of the enhanced image feature map, The number of parameters representing the mask feature, C d Represents the number of dimensions, H i , W i The i-th enhanced image feature map Fv is i The height and width of H, i represents the i-th scale, m , W m are the height and width of the m-th mask feature M respectively, and m represents the m-th scale.

[0093] Exemplarily, when the original image is a three-channel image I with a height and a width of H and W respectively, the image encoder G is used. v The image feature maps of four scales are extracted and fused to obtain the image feature maps of four scales with text information. After pixel decoding, the enhanced image feature maps Fv of three scales are obtained. i , the three scale enhanced image feature maps Fv i The height and width of are 1 / 32, 1 / 16, and 1 / 8 of the height H and width W of the original image I, respectively.

[0094] In this embodiment, Swin-Base, Swin-Tiny and Res-50 are used as the visual backbone network in the image feature extraction stage, and the influence of the fusion block and the pixel decoder on the model parameter quantity and running speed is measured. The results are as follows: Fig. 9 As shown,

[0095] When Swin-Base, Swin-Tiny and Res-50 are used as the feature extraction backbone networks, the number of parameters is approximately 211 million, 142 million, and 160 million, respectively.

[0096] It is understandable that in Swin-base and Swin-tiny, the parameters of the fusion block account for no more than 5% of the total parameters, while in Res-50, the parameters of the fusion block account for nearly 17%. This is because the feature dimension of the Res-50 backbone network is larger than that of the other two backbone networks.

[0097] In the three networks, the pixel decoder parameters account for no more than 2%. Considering the impact of the pixel decoder and fusion block on segmentation performance, these two modules only bring a small parameter burden and have considerable parameter efficiency.

[0098] S4. Query decoding is performed on the enhanced image feature maps of the multiple different scales using the mask feature and the text feature to obtain a segmentation map containing target information.

[0099] Preferably, the Query decoding uses a learnable Object Query decoder for combining image information and text information and generating information representing a prototype of a target object.

[0100] In an optional solution, in order to avoid the unstable Query-Target bipartite matching problem that may be caused by multiple Object Queries, the Query decoder adopts a single learnable Object Query 504, which interacts with the image and text information alternately, and updates itself by alternately querying the multi-scale image information and text information through multiple layers of Query decoder layers.

[0101] In this embodiment, if Figure 5 As shown, the Object Query decoder includes N decoding layers, each of which includes a text cross-attention module Attn t 501. Image Cross Attention Module Attn v 502 and feedforward neural network FFN 503, including FFN 503 consists of two linear layers with residual connections and non-linear activation functions.

[0102] The Object Query 504 represents a target object prototype having a certain feature, and is responsible for detecting / segmenting this type of object in the image during the decoding process.

[0103] Exemplarily, the Query decoding process includes:

[0104] A single learnable Object Query504 is randomly initialized using a standard normal distribution to represent the target object, denoted as Q 0 , in, Indicates the number of parameters of the Object Query. q Indicates the dimension of Object Query. 0 Enter the first decoding layer as output;

[0105] For the mth decoding layer (0<m<N), the text cross attention module Attn in the decoding layer t 501 receives the output Q of the previous decoding layer 0 As the query of the attention mechanism, F acts as the key and value, through the text cross attention module Attn t 501 interacts to generate a query carrying text information, recorded as The process can be expressed as:

[0106]

[0107] Among them, Q represents the query of the attention mechanism, K represents the key of the attention mechanism, Val represents the value of the attention mechanism, and C l Represents the number of text feature dimensions, Both are learnable weight matrices in the attention mechanism.

[0108] Said Before entering the image cross attention module Attn v Before 502, the mask feature M is first subjected to a dot multiplication segmentation process to obtain a phased output pred m In this embodiment, a simple point multiplication segmentation head is introduced to generate the segmentation map. The segmentation head consists of a 3-layer linear feedforward network MLP with a nonlinear activation function, and outputs pred in stages. m The generation process can be expressed as:

[0109]

[0110] The phased output pred m After downsampling and binarization (the cutoff value can be set to 0.5), we get M m , for M m Perform weighted masking operation to obtain Finally As the attention mask of the image cross-attention mechanism, for the i-th layer image attention cross-attention module, the input enhanced image feature map Fv i and The masked attention mechanism can be expressed as:

[0111]

[0112] Among them, MaskedAttn v m represents the attention mask of the mth layer, C l Represents the number of text feature dimensions,

[0113] Both are learnable weight matrices in the attention mechanism.

[0114] Understandable, attention mask It acts as a soft region of interest, which can exclude regions of no interest in advance to reduce noise interference. It is obtained by binarizing, downsampling, and tensor stretching the stage output. The value at the (x,y) position can be calculated using the following formula:

[0115]

[0116] Interactive processing with the mask feature M, the processed result is the same as the enhanced image feature map Fv i Sequentially through the image cross attention module Attn v 502 interacts with the feedforward neural network FFN503 to obtain Q m , the Q m As the output of the mth decoding layer, it enters the m+1th decoding layer;

[0117] For the Nth decoding layer, receive the output Q of the N-1th decoding layer N-1 , the Q N-1 The text features are combined with the text features through the text cross attention module Attn t 501 interacts to generate a Query carrying text information, the Query carrying text information is interactively processed with the mask feature M, and the processed result, the enhanced image feature map and the Query carrying text information are jointly processed by the image cross attention module Attn v 502 interacts, and the result after the interaction is enhanced with the nonlinear expression ability through the feed-forward neural network FFN, and then the result is subjected to point multiplication segmentation processing with the mask feature M to obtain a segmentation map containing target information.

[0118] In this embodiment, the Query decoder uses the weighted sum of the dice loss function and the cross entropy loss function ce as the final loss function. The calculation formula of the final loss function is expressed as:

[0119]

[0120] Where N is the number of layers of the Query decoder, pred jis the stage output of the j-th decoding layer, j represents the j-th decoding layer, target is the segmentation map containing target information, λ dice is the weight of the dice loss function, λ ce is the weight of the cross entropy loss function ce, dice_loss and ce_loss are the weighted results of the dice loss function and the cross entropy loss function ce respectively, and loss is the final loss function.

[0121] On the other hand, this embodiment provides a reference image segmentation system based on multi-level feature fusion, such as Figure 6 As shown, including,

[0122] The encoding module 601 includes a text encoding module and an image encoding module, wherein the text encoding module is used to encode text information into text features, and the image encoding module is used to encode the original image with target information into multiple image feature maps of different scales;

[0123] A fusion module 602 is used to fuse the image feature map of each scale with the text feature to generate a plurality of image feature maps of different scales that fuse text information;

[0124] A pixel decoding module 603 is used to perform pixel decoding on the image feature maps of multiple scales that incorporate text information to obtain mask features and enhanced image feature maps of multiple scales;

[0125] The query decoding module 604 is used to perform query decoding on the multiple enhanced image feature maps of different scales by using the mask features and the text features to obtain a segmentation map containing target information.

[0126] It can be understood that the above-mentioned system embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the system embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A reference image segmentation system based on multi-level feature fusion provided in an embodiment of the present application can execute a reference image segmentation method based on multi-level feature fusion provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the reference image segmentation system based on multi-level feature fusion can be implemented in the form of hardware, can be implemented by instructions in the form of software, and can also be implemented by a combination of hardware and software modules.

[0127] Optionally, the software module may be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0128] The present application embodiment provides an electronic device 710, whose structure is as follows: Figure 7 The electronic device 710 may be Figure 8 The server 100 or the terminal 200 is shown.

[0129] like Figure 7 As shown, the electronic device 710 includes a memory 711, a processor 712, a communication module 713 and an input / output interface 714, etc. Optionally, the memory 711, the processor 712, the communication module 713 and the input / output interface 714 can be connected and communicated through a bus 715.

[0130] The memory 711 is used to store one or more computer programs and transfer the code of the computer program to the processor 712; when the one or more computer programs are executed by the processor 711, a reference image segmentation method based on multi-level feature fusion in an embodiment of the present application is implemented.

[0131] Optionally, the electronic device 710 can be connected to a network via a communication module 713 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 710 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 710 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0132] Optionally, the electronic device 710 can be connected to a required input / output device, such as a keyboard, a display device, etc., through the input / output interface 714. The electronic device 710 itself can have a display device, and can also be connected to other external display devices through the input / output interface 714. Optionally, a storage device, such as a hard disk, can be connected through the input / output interface 714, so that data in the electronic device 710 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 711.

[0133] It is understandable that the input / output interface 714 may be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 714 may be a component of the electronic device 710 or an external device connected to the electronic device 710 when needed.

[0134] Optionally, the memory 711 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0135] Optionally, the computer program stored in the processor 711 may be divided into one or more modules, which are stored in the memory 711 and executed by the processor 712 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 710.

[0136] Optionally, the processor 712 may be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 712 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 712 executes various methods and processes of this embodiment, illustratively, such as a reference image segmentation method based on multi-level feature fusion in an embodiment of the present application.

[0137] Optionally, the bus 715 may include a path for transmitting information. The bus 715 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 715 may be divided into an address bus, a data bus, a control bus, etc.

[0138] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 711 of the electronic device 710. When the computer program is executed by the processor 712, one or more steps of a reference image segmentation method based on multi-level feature fusion in the embodiment of the present application can be performed.

[0139] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0140] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A reference image segmentation method based on multi-level feature fusion, characterized in that: include, Encode text information into text features, and encode the original image with target information into multiple image feature maps of different scales; The image feature map of each scale is fused with the text features respectively to generate multiple image feature maps of different scales that incorporate text information; The image feature maps of multiple scales that incorporate text information are decoded pixel by pixel to obtain mask features and enhanced image feature maps of multiple scales; The mask features and the text features are used to perform query decoding on the enhanced image feature maps of the multiple different scales to obtain a segmentation map containing target information.

2. The reference image segmentation method based on multi-level feature fusion according to claim 1, characterized in that: During the fusion process, the multiple image features of different scales are respectively subjected to an attention mechanism operation with the text features, and the obtained result is element-wise multiplied with the image feature projection result. The result after the element-wise multiplication is projected to obtain an image feature map of multiple scales that fuses the text information.

3. The reference image segmentation method based on multi-level feature fusion according to claim 2, characterized in that: The fusion process is specifically expressed by the following formula: Q q =V i W Q1 ,K k =FW K1 ,V v =FW V1 Where, subscript i represents the i-th scale, V i represents the image feature map of the i-th scale, F represents the text feature, and W Q1 , W Q2 , W K1 , W V1 , W F are all linear transformation matrices; Q q Represents the image feature map of the i-th scale and W Q1 The result after multiplication; K k Represents text features and W K1 The result after multiplication; V v Represents text features and W V1 The result after multiplication; ⊙ represents element multiplication; Softmax represents the normalization function; d k Indicates the dimension of the key vector, Fuse i (V i ,F) represents the image feature map of the i-th scale that integrates text information, and n represents the total number of scales.

4. The reference image segmentation method based on multi-level feature fusion according to claim 1, characterized in that: The encoding of the text information into text features specifically includes: Preset text information, and convert the text information into a feature sequence; Encoding the feature sequence to obtain text features; and / or, The encoding of the original image with target information into multiple image feature maps of different scales specifically includes: The original image with target information is encoded, and a feature map is extracted once in each encoding stage to obtain multiple image feature maps of different scales.

5. The reference image segmentation method based on multi-level feature fusion according to claim 1, characterized in that: The pixel decoding adopts a multi-scale deformable attention module and a feedforward neural network. Based on the attention module and the feedforward neural network, the image feature maps of multiple scales that are integrated with text information are decoded to interact between the feature maps of different scales to obtain the mask features and the enhanced image feature maps of the multiple different scales.

6. The reference image segmentation method based on multi-level feature fusion according to claim 1, characterized in that: The query decoding adopts a learnable object query decoder, and the object query decoder includes multiple decoding layers, each decoding layer includes a text cross attention module, an image cross attention module and a feedforward neural network FFN. The query decoding process includes: Assume that there are N decoding layers, initialize Query to represent the target information, and enter the first decoding layer as output; For the 1st to Nth decoding layers, the output of the previous decoding layer is received and interacted with the text feature through the text cross attention module to generate a query carrying text information. The query carrying text information and the mask feature are processed by point multiplication and segmentation to obtain an attention mask. The attention mask interacts with the enhanced image feature map and the query carrying text information through the image cross attention module. The result of the interaction is enhanced with nonlinear expression ability through a feedforward neural network FFN as output; For the Nth decoding layer, the result after interaction through the image cross attention module is enhanced with the nonlinear expression ability through the feedforward neural network FFN, and then processed with the mask feature through point multiplication segmentation to obtain a segmentation map containing target information.

7. The reference image segmentation method based on multi-level feature fusion according to claim 6, characterized in that: The query decoding uses the weighted sum of the dice loss function and the cross entropy loss function ce as the final loss function. The calculation formula of the final loss function is expressed as: Where N is the number of layers of the Query decoder, pred j is the stage output, j represents the jth decoding layer, target is the segmentation map containing target information, λ dice is the weight of the dice loss function, λ ce is the weight of the cross entropy loss function ce, dice_loss and ce_loss are the weighted results of the dice loss function and the cross entropy loss function ce respectively, and loss is the final loss function.

8. A reference image segmentation system based on multi-level feature fusion, characterized in that: It includes encoding module, pixel decoding module, query decoding module and fusion module; The encoding module includes a text encoding module and an image encoding module, the text encoding module is used to encode text information into text features, and the image encoding module is used to encode the original image with target information into multiple image feature maps of different scales; The fusion module is used to fuse the image feature map of each scale with the text feature respectively, to generate a plurality of image feature maps of different scales fused with text information; The pixel decoding module is used to perform pixel decoding on the image feature maps of multiple scales that incorporate text information, to obtain mask features and enhanced image feature maps of multiple scales; The query decoding module is used to perform query decoding on the multiple enhanced image feature maps of different scales using the mask features and the text features to obtain a segmentation map containing target information.

9. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements a reference image segmentation method based on multi-level feature fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to execute a reference image segmentation method based on multi-level feature fusion as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Lightweight crop disease image recognition method based on attention feature fusion

    CN115131662A

  • Image semantic segmentation method based on large convolution kernel backbone network

    CN116612283A

  • Creference image segmentation model training method and reference image segmentation method

    CN116993976A

  • Context-aware reference image segmentation method, system and device and storage medium

    CN117078942A

  • Remote sensing image segmentation method and system based on improved SwinTransform

    CN117095168A

Cited By

  • Image text segmentation method and device, electronic equipment and readable storage medium

    CN119339387A

  • Text-guided zero sample target counting method, program product and electronic equipment

    CN120147298A

  • Image segmentation method and electronic equipment

    CN121259322A

  • Image segmentation method and electronic device

    CN121259322B