A reference image segmentation method and system based on denoising mask transformer

Abstract

The present application relates to the field of image processing, and to a reference image segmentation method and system based on a denoising mask Transformer. The method includes: training and testing the model using a reference image with a real label, inputting the image to be processed into the segmentation model after training and testing for processing to obtain a segmentation map of the target information; the segmentation model training method includes: obtaining text features pointing to the target information, reference image features containing the target information, pixel decoding of the reference image features, obtaining mask features and enhanced reference image features, sampling from the real label to obtain a noisy real label, and performing DNMFormer decoding on the enhanced reference image features to obtain a segmentation map of the target information. Noisy real labels are used in the model training stage to provide accurate hard region suggestions, obtain appropriate feature information, and reduce the occurrence of harmful gradient updates; clustered DNM decoding and contextual DNM decoding are introduced to update the learnable query and quickly bridge the semantic gap between linguistic and visual modalities.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a reference image segmentation method and system based on a denoising mask Transformer. Background Art

[0002] Reference image segmentation is a cross-modal task that involves processing both visual and textual sensory information. It processes a given picture and text information pointing to an object in the picture, locates and segments the target object under the guidance of the text information, and generates a segmentation map of the area where the target object is located. To achieve a more ideal effect in this process, it is necessary to filter out irrelevant image features during the target image segmentation task.

[0003] Although existing technologies can predict the approximate position of reference objects and filter out irrelevant image features, unfavorable gradients in back-propagation can easily lead to erroneous filtering, and the use of global contextual attention leads to insufficient convergence speed. Summary of the invention

[0004] To solve at least one of the above-mentioned defects, the present invention application proposes a reference image segmentation method and system based on a denoising mask Transformer, which is used to reduce unfavorable gradients in back propagation and improve the convergence speed.

[0005] According to a first aspect of the present application, a reference image segmentation method based on a denoising mask Transformer is provided, comprising:

[0006] Acquire a reference image containing target information and text information pointing to the target information, divide the reference image containing target information into a training set and a test set, wherein the reference image has a true label;

[0007] Using the text information, training set and test set, training and testing the reference image segmentation model;

[0008] The image to be processed is input into a reference image segmentation model that has been trained and tested for processing to obtain a segmentation map of the target information.

[0009] A reference image containing target information is obtained according to target information in an image containing target information to be processed, and the reference image containing target information is divided into a training set and a test set, wherein the training set and the test set are respectively used for training and testing an image segmentation model.

[0010] The training method of the reference image segmentation model is as follows:

[0011] Encoding the text information pointing to the target information into text features, and encoding the reference image containing the target information in the training set into reference image features of multiple different scales;

[0012] Performing pixel decoding on the reference image features of the multiple different scales to obtain mask features and enhanced reference image features of the multiple different scales;

[0013] The noisy true labels are sampled from the true labels based on the Bernoulli distribution probability;

[0014] The enhanced reference image features of the multiple scales are decoded by DNMFormer using the noisy true label, the mask feature and the text feature to obtain a segmentation map of the target information.

[0015] Preferably, the sampling from the real label based on the Bernoulli distribution probability to obtain the noisy real label specifically includes:

[0016] Downsampling the true label in the reference image to the resolution of the reference image feature to obtain a downsampling result;

[0017] Generate a Bernoulli distribution probability according to the size relationship between the area of ​​the target information and the complete area of ​​the reference image and the downsampling result;

[0018] The Bernoulli distribution probability is used to sample the true labels to obtain noisy true labels.

[0019] Preferably, the sampling result is set to GT i (x, y), when the downsampling result is in the target object area, GT i (x, y) = 1, Bernoulli probability is λ1, when the downsampling result is in the background area, GT i (x, y) = 0, the Bernoulli probability is λ2,

[0020] Using λ1 as a hyperparameter, the Bernoulli probability is modified as follows:

[0021]

[0022]

[0023] Among them, S r and S represent the area of ​​the reference object and the area of ​​the entire image, respectively, and r is a hyperparameter for controlling the noise level.

[0024] The Bernoulli probability is expressed as,

[0025]

[0026] Based on the Bernoulli distribution probability, the true label is sampled to obtain a binary mask as the noisy true label.

[0027] Preferably, the DNMFormer decoding includes clustered DNM decoding, and the specific process is:

[0028] Assume that there are N decoding layers, each decoding layer includes multiple initial queries, and the initial queries include reference queries and other queries;

[0029] For the first decoding layer:

[0030] The reference query in the initial query is randomly flipped to obtain the flipped denoised query, which is specifically expressed as:

[0031]

[0032] in, represents the denoised Query of the first decoding layer, Represents the reference query of level 1, is a Gaussian random vector whose value follows a standard normal distribution and whose dimension is the dimension of the Query. p represents the preset probability of flipping the reference Query.

[0033] The enhanced reference image feature is matrix multiplied with the noisy true label in the reference image, and the result is matrix added with the flipped denoising query to obtain a first denoising query, and the first denoising query is used as the output denoising query of the first decoding layer and enters the second decoding layer;

[0034] For the 2nd-Nth decoding layers:

[0035] The reference query in the initial query is randomly flipped to obtain a flipped denoised query;

[0036] The enhanced reference image feature is matrix multiplied with the noisy true label in the reference image, and the obtained result is matrix added with the flipped denoising Query to obtain a first denoising Query;

[0037] Receive the output denoising query of the previous layer, use the output denoising query of the previous layer as the denoising query, perform matrix multiplication on the denoising query and the enhanced reference image feature to obtain a denoising attention map, perform matrix multiplication on other queries in the initial query and the enhanced reference image feature to obtain other attention maps, obtain a clustering result based on the denoising attention map and the other attention maps, perform matrix multiplication on the clustering result and the enhanced reference image feature, obtain a result and perform matrix addition on the denoising query to obtain a second denoising query, the second denoising query;

[0038] The first denoising query and the second denoising query are used together as the output denoising query of the current decoding layer and enter the next decoding layer;

[0039] For the 1st-Nth decoding layers:

[0040] The multiple initial queries are matrix multiplied with the enhanced reference image features, and the obtained results are subjected to clustered argmax processing to obtain argmax results. The argmax results are matrix multiplied with the enhanced reference image features of the decoding layer. The results of the matrix multiplication are matrix added with the initial queries to obtain a new initial query. The new initial query enters the next decoding layer as the initial query of the next decoding layer.

[0041] Optionally, the DNMFormer decoding includes the context DNM decoding, and the specific process is:

[0042] Assume that there are N decoding layers, and the initial Query of each decoding layer includes a reference Query;

[0043] In each decoding layer, the reference query is randomly flipped to obtain the denoised query, which is specifically expressed as:

[0044]

[0045] in, represents the denoised Query of the i-th decoding layer, represents the reference Query of the i-th layer, is a Gaussian random vector whose value follows a standard normal distribution and whose dimension is the dimension of Query, and p represents the predetermined probability of flipping the reference Query. It can be understood that when the decoding layer is not the first decoding layer, the denoised Query includes the output denoised Query from the previous decoding layer in addition to the denoised Query obtained by flipping the current decoding layer.

[0046] The denoised query is matrix multiplied with the enhanced reference image features, and the result is matrix added and softmax processed with the noisy true label to obtain a softmax result. The softmax result is matrix multiplied with the enhanced reference image features, and the result is matrix added with the denoised query to obtain a new denoised query. The new denoised query output denoised query of this decoding layer enters the next decoding layer as the denoised query of the next decoding layer.

[0047] Preferably, after the last decoding layer of DNMFormer decoding, a mask terminal MLP is introduced to receive and process the reference query and denoised query outputted by the last decoding layer to obtain the segmentation probability Z, which is specifically expressed as:

[0048]

[0049] Z=Sigmoid(T Q M)

[0050] in represents the denoised query and reference query output by the last decoding layer of DNMFormer, T Q represents the denoised Query and reference Query after linear transformation, M represents the mask feature, and Sigmoid() represents the normalization function.

[0051] The segmentation probability is used to reflect the pixel category, and a segmentation map is obtained by binarizing the segmentation probability.

[0052] Preferably, the text information pointing to the target information is encoded as text features, specifically including:

[0053] Preset a text information pointing to the target information, and convert the text information pointing to the target information into a feature sequence;

[0054] Encode the feature sequence to obtain text features;

[0055] and / or,

[0056] The reference image containing target information in the training set is encoded into a plurality of reference image features of different scales, specifically including:

[0057] The reference images containing target information in the training set are encoded, and a feature map is extracted once in each encoding stage to obtain reference image features of multiple different scales.

[0058] In an optional solution, a word segmenter of a text encoder is used to convert the text information into a feature sequence. Specifically, BERT with the last pooling layer removed can be used as the text encoder.

[0059] An image encoder is used to encode the reference image containing target information. Specifically, SwinTransformer can be selected as the image encoder.

[0060] Preferably, the pixel decoding uses a multi-scale deformable attention module and a feed-forward neural network to decode image features of multiple scales to interact between feature maps of different scales to obtain enhanced reference image features of the multiple scales.

[0061] Optionally, the DNMFormer decoder uses a weighted sum of a dice loss function and a cross entropy loss function ce as the final loss function, and the calculation formula of the final loss function is expressed as:

[0062]

[0063] Among them, target is the real label, λ dice is the weight of the dice loss function, λ ce is the weight of the cross entropy loss function ce, dice loss and loss They are the weighted results of the dice loss function and the cross entropy loss function ce, and loss is the final loss function.

[0064] According to a second aspect of the present application, a reference image segmentation device based on a denoising mask Transformer is provided, comprising an acquisition module, a training and testing module, and a segmentation processing module;

[0065] The acquisition module is used to acquire reference images containing target information and text information pointing to the target information, and divide the reference images containing target information into a training set and a test set, wherein the reference images have real labels;

[0066] The training and testing module is used to train and test the reference image segmentation model using the text information, training set and test set;

[0067] The segmentation processing module is used to input the image to be processed into a reference image segmentation model that has been trained and tested to obtain a segmentation map of the target information.

[0068] According to a third aspect of the present application, an electronic device is provided, including:

[0069] a memory for storing one or more computer programs;

[0070] The processor implements a reference image segmentation method based on a denoising mask Transformer according to the first aspect when the one or more computer programs are executed by the processor.

[0071] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a reference image segmentation method based on a denoising mask Transformer according to the first aspect when executed.

[0072] Based on any of the above aspects, the embodiments of the present application provide a reference image segmentation method and system based on a denoising mask transformer. In the segmentation model training stage, denoising true labels are used to provide accurate hard area suggestions, obtain appropriate feature information, and reduce the occurrence of harmful gradient updates; clustered DNM decoding and contextual DNM decoding are introduced to update learnable queries and quickly bridge the semantic gap between linguistic and visual modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 A flowchart of a reference image segmentation method based on a denoising mask Transformer provided in this embodiment.

[0075] Figure 2 A flowchart of a method for training a reference image segmentation model provided in this embodiment

[0076] Figure 3 An overall framework diagram of a reference image segmentation method based on a denoising mask Transformer provided in this embodiment.

[0077] Figure 4 A schematic diagram of the functional modules of a reference image segmentation device based on a denoising mask Transformer provided in this embodiment.

[0078] Figure 5 Schematic diagram of the clustered DNM decoding (left) and contextual DNM decoding (right) processes provided for this embodiment.

[0079] Figure 6 A schematic diagram of the internal structure of a DNMFormer decoder provided in this embodiment.

[0080] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this embodiment.

[0081] Figure 8 A schematic application scenario diagram of a reference image segmentation method based on a denoising mask Transformer provided in this embodiment.

[0082] Icons: 100, server; 200, terminal; 401, acquisition module; 402, training and testing module; 403, segmentation processing module; 710, electronic device; 711, memory; 712, processor; 713, communication module; 714, input / output interface; 715, bus. DETAILED DESCRIPTION

[0083] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0084] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0085] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0086] Although existing technologies can predict the approximate position of reference objects and filter out irrelevant pixel features, unfavorable gradients in back-propagation can easily lead to erroneous filtering, and the use of global contextual attention leads to insufficient convergence speed.

[0087] On the one hand, the embodiment of the present application provides a reference image segmentation method based on a denoising mask Transformer, such as Figure 1 As shown, the steps at least include,

[0088] S1, obtaining a reference image containing target information and text information pointing to the target information, dividing the reference image containing target information into a training set and a test set, wherein the reference image has a real label;

[0089] S2. Training and testing the reference image segmentation model using the text information, training set and test set;

[0090] S3, inputting the image to be processed into a reference image segmentation model that has been trained and tested for processing to obtain a segmentation map of the target information;

[0091] In this embodiment, the reference image segmentation model is trained using the following method:

[0092] A1. Encoding the text information pointing to the target information into text features, and encoding the reference image containing the target information in the training set into reference image features of multiple different scales;

[0093] A2. Pixel decoding is performed on the reference image features of the multiple different scales to obtain mask features and enhanced reference image features of the multiple different scales;

[0094] A3. Sample the true label with noise from the true label based on the Bernoulli distribution probability;

[0095] A4. Perform DNMFormer decoding on the enhanced reference image features of the multiple different scales using the noisy true label, the mask feature and the text feature to obtain a segmentation map of the target information.

[0096] The following is a combination of specific embodiments and Figure 2 , the technical solution provided in this application is described in detail.

[0097] A1. Encoding the text information pointing to the target information into text features, and encoding the reference image containing the target information in the training set into reference image features of multiple different scales;

[0098] In this embodiment, the text encoder G is used t Encode the text information into text features F and use the image encoder G v , encode the original image with target information into multiple reference image features V of different scales, where p represents the total number of scales.

[0099] Optionally, Bert-base with the last pooling layer removed is selected as the text encoder G in the embodiment of the present application. t In this embodiment, the text encoding process may include:

[0100] Preset text information T, use text encoder G tThe word segmenter converts it into a token sequence, truncates or fills it, and obtains a token sequence e with a fixed length of L. Then the token sequence e is sent to the text encoder G t Get the text feature F, The encoding process can be expressed as:

[0101]

[0102] Where L is the sequence length, C l Represents the number of text feature dimensions, The parameter quantity representing the text features.

[0103] Optionally, the image encoder G v Different mainstream image encoders can be used as visual backbone networks, such as Res-50, Swin-Tiny, and Swin-Base.

[0104] Exemplarily, in this embodiment, Swin Transformer is used as an image encoder, and the image encoding process includes:

[0105] For a three-channel image with height and width H and W respectively Using the image encoder G v Extract the reference image features V at four scales i , the process can be expressed as:

[0106]

[0107] Among them, V represents the feature map set, V i represents the reference image feature of the i-th scale, H i and W i Represent the height and width of the i-th scale reference image feature, I i represents the reference image feature of the i-th size, Represents the image feature dimension of the i-th image channel, i represents the i-th scale, the height and width of the four scale feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the height and width of the original image, respectively. Each scale feature map is extracted from an encoding stage.

[0108] A2. Perform pixel decoding on the reference image features of the multiple different scales to obtain mask features and enhanced reference image features of the multiple different scales.

[0109] In an optional implementation, a Multi-Scale Deformable with a multi-scale deformable attention mechanism can be used as the pixel decoder G p, which is used to model the interaction between feature maps of different scales, convey visual features at different levels, and generate high-resolution high-semantic mask features. p It comprises multiple pixel decoding layers, each of which comprises a multi-scale deformable attention module and a feedforward neural network with two layers of nonlinear activation functions. The pixel decoder G p Decode the reference image features of multiple scales that incorporate text information to obtain mask features and enhanced reference image features of multiple scales.

[0110] Exemplarily, when the original image is a three-channel image I with a height and a width of H and W respectively, the image encoder G is used. v Extract the reference image features V at four scales i ;

[0111] Pixel Decoder G p Receive image features V at four scales i , enhance the features V2, V3, and V4, which are 1 / 8, 1 / 16, and 1 / 32 of the original image size, to obtain the enhanced reference image feature V′={V′1, V′2, V′3}, Its scale sizes correspond to V2, V3, and V4 respectively. After that, it outputs a mask feature M with the same scale size as V1. The mask feature output by the pixel decoder and the enhanced reference image feature have the same channel dimension C.

[0112] in, represents the parameter quantity of the enhanced reference image feature, C represents the number of dimensions, and H i , W i They are the enhanced reference image features V′ i height and width.

[0113] A3. Sample the true label with noise from the true label based on the Bernoulli distribution probability;

[0114] Downsample the true label to the enhanced reference image feature V′ i The resolution of GT is used to obtain the down-sampling result. i express;

[0115] According to the area S of the reference object r The relationship between the size of the complete area S of the reference image and the probability of generating a Bernoulli distribution with the downsampling result When GT i When in the target information area, GT i (x, y) = 1, Bernoulli distribution probability is λ1, when GT i In the background area, GT i (x, y) = 0, Bernoulli distribution probability is λ2,

[0116] In order to reflect the influence of the reference object size, it is necessary to change λ1 or λ2 according to the target information area, and use λ1 as a hyperparameter to correct the Bernoulli distribution probability as follows:

[0117]

[0118]

[0119] Among them, r is a hyperparameter that controls the noise level.

[0120] The Bernoulli probability It is expressed as,

[0121]

[0122] Based on the Bernoulli distribution probability Use Bernoulli sampling to get a binary mask H and W represent the corresponding enhanced reference image features V′ i The height and width of the binary mask as the noisy true label.

[0123] A4. Perform DNMFormer decoding on the enhanced reference image features of the multiple different scales using the noisy true label, the mask feature and the text feature to obtain a segmentation map of the target information.

[0124] In this embodiment, a DNMFormer decoder is used for DNMFormer decoding, and the DNMFormer decoder includes multiple decoding layers, each decoding layer includes a QT-CA module, a DNM-attention module, a feedforward neural network, and two residual connections and normalization layers.

[0125] In an optional solution, the DNMFormer decoding may adopt clustered DNM decoding, and the specific process is as follows:

[0126] Assume that there are N decoding layers, each decoding layer includes multiple initial queries, and the initial queries include reference queries and other queries;

[0127] In the first decoding layer:

[0128] The reference query in the initial query of the first decoding layer is randomly flipped to obtain the flipped denoised query, which is specifically expressed as:

[0129]

[0130] in, represents the flipped denoising Query of the first decoding layer, Represents the reference query of level 1, is a Gaussian random vector whose value obeys the standard normal distribution and whose dimension is the dimension of Query. p represents the preset probability of flipping the reference Query, which is set to 0.5 in this embodiment.

[0131] The enhanced reference image feature V′1 and the noisy true label Perform matrix multiplication and obtain the result of the flip Perform matrix addition to obtain the first denoising query, which is used as the output denoising query of the first decoding layer, denoted as Said Entering the second decoding layer, it is specifically expressed as:

[0132]

[0133] in, represents the first denoising Query generated by the first decoding layer, V′1 represents the enhanced reference image features of the first scale, represents the noisy true label in the first decoding layer, W v Represents a linear transformation matrix.

[0134] In the i-th decoding layer, (1<i),

[0135] The reference query in the initial query is randomly flipped to obtain the flipped denoised query, which is specifically expressed as:

[0136]

[0137] in, represents the flipped denoised Query of the i-th decoding layer, represents the reference Query of the i-th decoding layer, is a Gaussian random vector whose value obeys the standard normal distribution and whose dimension is the dimension of Query. p represents a preset probability of flipping the reference Query, which is set to 0.5 in this embodiment.

[0138] The enhanced reference image feature V′ i and the noisy true label in the reference image Perform matrix multiplication, and perform matrix addition on the result obtained with the flip denoising Query to obtain the first denoising Query, which is specifically expressed as:

[0139]

[0140] in, represents the flip denoising Query of the i-th layer, Represents the first denoised Query.

[0141] receiving an output denoising query of the i-1th layer, wherein the output denoising query of the i-1th layer is used as the denoising query;

[0142] Then, the second denoising query is generated, which is specifically expressed as:

[0143]

[0144]

[0145]

[0146]

[0147] in, represents the second denoising Query, represents the output denoising Query of the i-1th layer, Represents other queries of the i-th decoding layer, V′ i represents the enhanced reference image features, represents the denoising attention map, represents other attention maps, represents the clustering result, represents the maximum value in other attention maps, W q , W k , W v Represents three linear transformation matrices.

[0148] Will and the enhanced reference image feature V′ i Perform matrix multiplication to obtain the denoised attention map Q i In and the enhanced reference image feature V′ i Perform matrix multiplication to get other attention maps Based on the denoising attention map and the other attention maps Get clustering results The clustering results With the enhanced reference image feature V′ i Perform matrix multiplication and get the result and Perform matrix addition to get

[0149] The first denoising query and the second denoising query are used together as the output denoising query of the current decoding layer and enter the next decoding layer;

[0150] The calculation process of reference query and other queries is expressed as:

[0151]

[0152] in, represents the reference query and other queries of the i-th decoding layer, Q {r，o} i+1 Denotes the reference query and other queries of the i+1th decoding layer, V′ i represents the enhanced reference image features, W q , W k , W v Represents three linear transformation matrices.

[0153] Multiple and the enhanced reference image feature V′ i The matrix multiplication is performed, and the result is processed by clustered argmax to obtain the argmax result, which is compared with the enhanced reference image feature V′ i Perform matrix multiplication, and then multiply the result of the matrix multiplication with Add the matrices to get Q {r，o} i+1 , the Q {r，o} i+1 As the output of the i-th decoding layer, it enters the i+1-th decoding layer and serves as the initial Query of the i+1-th decoding layer.

[0154] In another optional solution, the DNMFormer decoding may adopt contextual DNM decoding, and the specific process is as follows:

[0155] Assume that there are N decoding layers, and the initial Query of each decoding layer includes a reference Query;

[0156] For the i-th decoding layer (1≤i), the reference query in the initial query is randomly flipped to obtain the denoised query, which is specifically expressed as:

[0157]

[0158] in, represents the denoised Query of the i-th decoding layer, represents the reference Query of the i-th layer, is a Gaussian random vector whose value follows a standard normal distribution and whose dimension is the dimension of the Query. p represents the preset probability of flipping the reference Query.

[0159] It can be understood that when 1<i, the denoised query includes not only the denoised query obtained by flipping in the i-th decoding layer, but also the output denoised query from the i-1-th decoding layer.

[0160] Combine the denoised Query with the enhanced reference image feature V′ i Perform matrix multiplication and sum the result with the weighted masked noisy true label After matrix addition and softmax processing, a softmax result is obtained, and the softmax result and the enhanced reference image feature V′ i Perform matrix multiplication and add the result to the Perform matrix addition to get Said As the output denoising Query of the i-th decoding layer, it enters the i+1-th decoding layer, which is specifically expressed as:

[0161]

[0162]

[0163] in, is the denoising Query of the i-th decoding layer, is the denoised Query of the i+1th decoding layer, V′ i To enhance the reference image features, W q , W k , W v are three linear transformation matrices, is the true label with noise, is the weighted masked noisy true label.

[0164] In this embodiment, after the last decoding layer of DNMFormer decoding, a mask terminal MLP is introduced to receive and process the reference query and denoised query output by the last decoding layer to obtain the segmentation probability Z, which is specifically expressed as:

[0165]

[0166] Z=Sigmoid(T Q M)

[0167] in represents the denoised query and reference query output by the last decoding layer of DNMFormer, TQ represents the denoised Query and reference Query after linear transformation, M represents the mask feature, and Sigmoid() represents the normalization function.

[0168] The segmentation probability is used to reflect the pixel category, and the segmentation map is obtained after the segmentation probability is binarized (the threshold is set to 0.5).

[0169] In this embodiment, the DNMFormer decoder uses the weighted sum of the dice loss function and the cross entropy loss function ce as the final loss function. The calculation formula of the final loss function is expressed as:

[0170]

[0171] Among them, target is the real label, λ dice is the weight of the dice loss function, λ ce is the weight of the cross entropy loss function ce, dice loss and loss They are the weighted results of the dice loss function and the cross entropy loss function ce, and loss is the final loss function.

[0172] On the other hand, this embodiment provides a reference image segmentation device based on a denoising mask Transformer, such as Figure 4 As shown, including,

[0173] An acquisition module 401 is used to acquire a reference image containing target information and text information pointing to the target information, and divide the reference image containing target information into a training set and a test set, wherein the reference image has a real label;

[0174] A training and testing module 402, used to train and test the reference image segmentation model using the text information, training set and test set;

[0175] The segmentation processing module 403 is used to input the image to be processed into a reference image segmentation model that has been trained and tested for processing to obtain a segmentation map of the target information.

[0176] It can be understood that the above-mentioned system embodiment and the above-mentioned method embodiment can correspond to each other, and similar descriptions of the system embodiment can refer to the method embodiment. To avoid repetition, it will not be repeated here. A reference image segmentation system based on a denoising mask Transformer provided in an embodiment of the present application can execute a reference image segmentation method based on a denoising mask Transformer provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method. The functional modules of the reference image segmentation system based on multi-level feature fusion can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software modules.

[0177] Optionally, the software module may be located in a random access memory, a read-only memory, a programmable read-only memory, a flash memory, an electrically erasable programmable memory, a register, or other storage media. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in combination with its hardware.

[0178] The present application embodiment provides an electronic device 710, whose structure is as follows: Figure 7 The electronic device 710 may be Figure 8 The server 100 or the terminal 200 is shown.

[0179] like Figure 7 As shown, the electronic device 710 includes a memory 711, a processor 712, a communication module 713 and an input / output interface 714, etc. Optionally, the memory 711, the processor 712, the communication module 713 and the input / output interface 714 can be connected and communicated through a bus 715.

[0180] The memory 711 is used to store one or more computer programs and transmit the code of the computer program to the processor 712; when the one or more computer programs are executed by the processor 711, a reference image segmentation method based on a denoising mask Transformer in an embodiment of the present application is implemented.

[0181] Optionally, the electronic device 710 can be connected to a network via a communication module 713 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device 710 can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device 710 can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0182] Optionally, the electronic device 710 can be connected to a required input / output device, such as a keyboard, a display device, etc., through the input / output interface 714. The electronic device 710 itself can have a display device, and can also be connected to other external display devices through the input / output interface 714. Optionally, a storage device, such as a hard disk, can be connected through the input / output interface 714, so that data in the electronic device 710 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 711.

[0183] It is understandable that the input / output interface 714 may be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 714 may be a component of the electronic device 710 or an external device connected to the electronic device 710 when needed.

[0184] Optionally, the memory 711 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0185] Optionally, the computer program stored in the processor 711 may be divided into one or more modules, which are stored in the memory 711 and executed by the processor 712 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device 710.

[0186] Optionally, the processor 712 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 712 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special artificial intelligence computing chips, various processors running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 712 executes the various methods and processes of this embodiment, illustratively, such as a reference image segmentation method based on a denoising mask Transformer in an embodiment of the present application.

[0187] Optionally, the bus 715 may include a path for transmitting information. The bus 715 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 715 may be divided into an address bus, a data bus, a control bus, etc.

[0188] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 711 of the electronic device 710. When the computer program is executed by the processor 712, one or more steps of a reference image segmentation method based on a denoising mask transformer in the embodiment of the present application can be executed.

[0189] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0190] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A reference image segmentation method based on denoising mask Transformer, characterized in that: The method comprises, Acquire a reference image containing target information and text information pointing to the target information, divide the reference image containing target information into a training set and a test set, wherein the reference image has a true label; Using the text information, training set and test set, training and testing the reference image segmentation model; Inputting the image to be processed into a reference image segmentation model that has been trained and tested for processing to obtain a segmentation map of the target information; The training method of the reference image segmentation model is as follows: Encoding the text information pointing to the target information into text features, and encoding the reference image containing the target information in the training set into reference image features of multiple different scales; Performing pixel decoding on the reference image features of the multiple different scales to obtain mask features and enhanced reference image features of the multiple different scales; The noisy true label is obtained by sampling from the true label based on the Bernoulli distribution probability; The enhanced reference image features of the multiple scales are decoded by DNMFormer using the noisy true label, the mask feature and the text feature to obtain a segmentation map of the target information.

2. The reference image segmentation method based on denoising mask Transformer according to claim 1, characterized in that: The method of sampling the true label with noise from the true label based on the Bernoulli distribution probability specifically includes: Downsampling the true label in the reference image to the resolution of the reference image feature to obtain a downsampling result; Generate a Bernoulli distribution probability according to the size relationship between the area of ​​the target information and the complete area of ​​the reference image and the downsampling result; The Bernoulli distribution probability is used to sample the true labels to obtain noisy true labels.

3. The reference image segmentation method based on denoising mask Transformer according to claim 2, characterized in that: The specific generation process of the noisy true label is expressed as: Let the sampling result be GT i (x,y), when the downsampling result is in the target object area, GT i (x,y)=1, Bernoulli probability is λ1, when the downsampling result is in the background area, GT i (x,y)=0, Bernoulli probability is λ2, Using λ1 as a hyperparameter, the Bernoulli probability is modified as follows: Among them, S r and S represent the area of ​​the reference object and the area of ​​the entire image, respectively, and r is a hyperparameter that controls the noise level; The Bernoulli probability is expressed as, Based on the Bernoulli distribution probability, the true label is sampled to obtain a binary mask as the noisy true label.

4. The reference image segmentation method based on denoising mask Transformer according to claim 1, characterized in that: The DNMFormer decoding includes clustered DNM decoding, and the specific process is as follows: Assume that there are N decoding layers, each decoding layer includes multiple initial queries, and the initial queries include reference queries and other queries; For the first decoding layer, the reference query in the initial query is randomly flipped to obtain the flipped denoised query; The enhanced reference image feature is matrix multiplied with the noisy true label in the reference image, and the result is matrix added with the flipped denoising query to obtain a first denoising query, and the first denoising query is used as the output denoising query of the first solution layer and enters the second decoding layer; For the 2nd-Nth decoding layers: The reference query in the initial query is randomly flipped to obtain a flipped denoised query; The enhanced reference image feature is matrix multiplied with the noisy true label in the reference image, and the obtained result is matrix added with the flipped denoising query to obtain a first denoising query; Receive the output denoising query of the previous layer, use the output denoising query of the previous layer as the denoising query, perform matrix multiplication on the denoising query and the enhanced reference image feature to obtain a denoising attention map, perform matrix multiplication on other queries in the initial query and the enhanced reference image feature to obtain other attention maps, obtain a clustering result based on the denoising attention map and the other attention maps, perform matrix multiplication on the clustering result and the enhanced reference image feature, and perform matrix addition on the result and the denoising query to obtain a second denoising query; The first denoising query and the second denoising query are used together as the output denoising query of the current decoding layer and enter the next decoding layer; For the 1st-Nth decoding layers: Perform matrix multiplication on the multiple initial queries and the enhanced reference image features, and obtain the results through clustered argmax processing to obtain argmax results. Perform matrix multiplication on the argmax results and the enhanced reference image features of the decoding layer. Perform matrix addition on the results of the matrix multiplication and the initial query to obtain a new initial query. The new initial query enters the next decoding layer as the initial query of the next decoding layer. The DNMFormer decoding includes contextual DNM decoding, and the specific process is as follows: Assume that there are N decoding layers, each decoding layer includes multiple initial queries, and the initial queries include reference queries; In each decoding layer, the reference query in the initial query is randomly flipped to obtain the denoised query; When the decoding layer is not the first decoding layer, the denoising query also includes the denoising query from the previous decoding layer; The denoised query is matrix multiplied with the enhanced reference image features, and the result obtained is matrix added and softmax processed with the noisy true label to obtain a softmax result. The softmax result is matrix multiplied with the enhanced reference image features, and the result obtained is matrix added with the denoised query to obtain a new denoised query, and the new denoised query enters the next decoding layer.

5. The reference image segmentation method based on denoising mask Transformer according to claim 4, characterized in that: The method further comprises: The mask feature and the reference query and denoised query output by the last DNMFormer decoding layer are received by the mask terminal and processed to obtain a segmentation probability matrix, and the matrix is ​​binarized to obtain a segmentation map of the target information.

6. The reference image segmentation method based on denoising mask Transformer according to claim 1, characterized in that: Encoding the text information pointing to the target information into text features, specifically including: Preset a text information pointing to the target information, and convert the text information pointing to the target information into a feature sequence; Encode the feature sequence to obtain text features; and / or, The reference image containing target information in the training set is encoded into a plurality of reference image features of different scales, specifically including: The reference images containing target information in the training set are encoded, and a feature map is extracted once in each encoding stage to obtain reference image features of multiple different scales.

7. The reference image segmentation method based on denoising mask Transformer according to claim 1, characterized in that: The pixel decoding adopts a multi-scale deformable attention module and a feed-forward neural network to decode reference image features of multiple scales to interact between feature maps of different scales and obtain enhanced reference image features of the multiple scales.

8. A reference image segmentation device based on a denoising mask Transformer, characterized in that: include, An acquisition module, used to acquire a reference image containing target information and text information pointing to the target information, and divide the reference image containing target information into a training set and a test set, wherein the reference image has a real label; A training and testing module, used to train and test the reference image segmentation model using the text information, training set and test set; A segmentation processing module, used for inputting the image to be processed into a reference image segmentation model that has been trained and tested to obtain a segmentation map of the target information; The training of the reference image segmentation model includes: Encoding the text information pointing to the target information into text features, and encoding the reference image containing the target information in the training set into reference image features of multiple different scales; Performing pixel decoding on the reference image features of the multiple different scales to obtain mask features and enhanced reference image features of the multiple different scales; The noisy true label is obtained by sampling from the true label based on the Bernoulli distribution probability; The enhanced reference image features of the multiple scales are decoded by DNMFormer using the noisy true label, the mask feature and the text feature to obtain a segmentation map of the target information.

9. An electronic device, characterized in that: include: a memory for storing one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements a reference image segmentation method based on a denoising mask Transformer as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to execute a reference image segmentation method based on a denoising mask Transformer as described in any one of claims 1 to 7.

Images