Multi-target image segmentation method based on cross-modal deep learning

Abstract

The invention discloses a multi-target image segmentation method based on cross-modal deep learning. The method comprises the following steps: (1) acquiring cucumber image data and text data and carrying out preprocessing; (2) extracting features of an input image by using an encoder in DeepLab v3 based on a convolutional neural network model, and extracting features of an input text by using a long-short term memory (LSTM) network of a recurrent neural network model; (3) selecting an unsupervised mask generator FreeSOLO to obtain mask candidate areas of all targets in the input image, and adopting element-by-element multiplication to obtain visual features of each mask; (4) processing the mask visual features and the text features to obtain a corresponding feature sequence, and calculating the similarity between the image sequence and the text sequence by adopting a cross-modal attention mechanism to obtain an attention weight; (5) setting an activation gate behind the fusion mask features output by the attention mechanism; the activated mask candidate areas are screened out through a filter and aggregated through an aggregator; (6) fusing the masks obtained by aggregation and the extracted visual features, and inputting the fused masks and the extracted visual features into a long-short term memory network for text reconstruction; and (7) outputting a result. According to the invention, an accurate multi-target segmentation result can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a multi-target image segmentation method based on cross-modal deep learning. Background Art

[0002] The multi-target image segmentation task mainly involves the following four technologies: First, image feature extraction is one of the basic tasks of image segmentation, which aims to extract features that are helpful for segmentation from the input image. Among the current deep learning methods, convolutional neural networks, dilated convolutions, visual transformers, etc. can effectively obtain image information of different scales through the above methods. They have outstanding performance in image feature extraction. Second, text feature extraction also occupies an important position in multimodal segmentation tasks, especially in scenarios combined with natural language descriptions. Word embedding technology, BERT and Transformer architectures, CLIP models and other technologies are widely used in text feature extraction, which can effectively capture contextual information to help segmentation models accurately identify and distinguish different target objects. Third, multimodal feature fusion is a key step in the multi-target reference image segmentation task. Feature fusion through attention mechanism and contrastive learning is currently a relatively mature and effective method. Fourth, using visual features to reconstruct text is a strong guarantee in multi-target reference image segmentation. The classic recurrent neural network model LSTM is currently widely used for implementation. The main problems in the current task are as follows: the background complexity of image data and the diversity of cucumber target shapes; the segmentation performance is poor, with problems such as incorrect segmentation or low segmentation accuracy; the computational complexity of the task is high and cannot meet real-time processing requirements. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a multi-target image segmentation method based on cross-modal deep learning to solve the problems of poor image segmentation performance, erroneous segmentation or low segmentation accuracy;.

[0004] Technical solution: The multi-target image segmentation method based on cross-modal deep learning described in the present invention comprises the following steps:

[0005] (1) Collecting cucumber image data including color, shape, length, thickness and text data; and performing preprocessing;

[0006] (2) The encoder in DeepLab v3 based on the convolutional neural network model is used to extract the features of the input image, and the recurrent neural network model long short-term memory network LSTM is used to extract the features of the input text;

[0007] (3) The unsupervised mask generator FreeSOLO is selected to obtain the mask candidate regions of all targets in the input image, and the two-dimensional binary matrix corresponding to the mask is fused with the visual features after dimensionality reduction by element-by-element multiplication to obtain the visual features of each mask;

[0008] (4) Processing the masked visual features and the text features in step (2) to obtain the corresponding feature sequence, using the cross-modal attention mechanism to calculate the similarity between the image sequence and the text sequence to obtain the attention weight, performing weighted calculation on the visual feature vector, and outputting the masked visual features that incorporate the text features;

[0009] (5) Setting an “activation gate” after the fused mask features output by the attention mechanism; using filters to filter out activated mask candidate regions and using an aggregator to aggregate them;

[0010] (6) The mask obtained by aggregation is fused with the visual features extracted in step (2), and input into the long short-term memory network for text reconstruction. The Euclidean distance between the reconstructed text features and the input text features is calculated as the consistency loss;

[0011] (7) Adjust the attention weight of the cross-modal fusion part and the activation gate part for the activation selection of the mask candidate area until the consistency loss is lower than the threshold or the number of iterations reaches the preset maximum value.

[0012] Furthermore, in step (1), the preprocessing includes: denoising, adjusting the image data to a uniform size, and dividing the data set into a training set, a validation set, and a test set.

[0013] Furthermore, in step (2), feature extraction is performed on the input image to extract the features of the input image as visual features. The formula is as follows:

[0014] V=CNN(I)={v1, v2,...,v hw}

[0015] Where V∈R h×w×Cv is the feature map of the image, h and w are the height and width of the feature map respectively. In the feature map V, each feature vector v i ∈R Cv Represents the visual features of the i-th region in image I;

[0016] The long short-term memory network LSTM is used to encode the features of the input text query Q. The formula is as follows:

[0017] x=RNN(W e Q)

[0018] in, is a word embedding matrix, For the feature vector of the entire language query Q, the hidden state of the last step that encodes the entire query information is used as the query feature vector.

[0019] Furthermore, step (3) includes the following steps:

[0020] (31) FreeSOLO is selected to obtain a set of masks m of the target in the input image I, m∈M(I), that is, a set of two-dimensional binarization matrices with the same size as the input image;

[0021] (32) Downsample the object mask from 224×224 to a spatial size that matches the visual feature map;

[0022] (33) Expand the binary matrix corresponding to each mask and multiply it element-by-element with the visual feature to obtain the visual feature corresponding to each mask:

[0023] V mi =V'⊙m'

[0024] Among them, V mi Represents the visual feature corresponding to the mask, i∈{1, 2, …, n}, n represents the total number of masks. ⊙ represents the Hadamard product, also known as the element-wise product.

[0025] Further, step (4) includes the following steps:

[0026] (41) Use two linear layers to map visual and textual features to the same dimension;

[0027] (42) The cross-modal attention mechanism is used to capture the correlation between visual features and text features, and perform feature fusion through the calculation of query, key, and value;

[0028] (43) Encode text features as Query, encode visual features as Key and Value, and use dot product to calculate the similarity between text and visual features;

[0029] (44) The similarities of all positions are normalized by the softmax function to generate attention weights. The visual features are weighted and summed using the attention weights to obtain the visual features associated with the text. The formula is as follows:

[0030]

[0031] Among them, softmax represents the activation function, Q, K, and V represent the query, key, and value encoded by visual and text features, respectively. k The dimension of Embedding.

[0032] Furthermore, in step (5), the “activation gate” is an n-dimensional column vector, in which each element is either 0 or 1, and the number of elements is the same as the number of mask candidate regions in step (3), including the following steps:

[0033] (51) Based on the visual features output by the attention mechanism, the activation mask is selected using the “activation gate”. The “activation gate” is essentially a 1×n column vector. When the corresponding element is 1, it indicates activation, and when the corresponding element is 0, it indicates inactivation.

[0034] (52) Using a filter to filter the activation mask to the aggregator layer, a union operation is used to achieve the purpose of retaining the area covered by all mask schemes, and a bitwise logical "OR" operation is performed on each mask area;

[0035] (53) For each pixel position, if the value of any mask scheme at that position is 1 or a valid value, the final result is 1; the formula is as follows:

[0036] FinalMask(i,j)=Mask1(i,j)∨Mask2(i,j)∨……∨Mask m (i,j)

[0037] Where (i, j) represents the pixel position, m is the total number of activated mask candidate regions, and finally the union m of all mask candidate regions will be obtained. f , retaining the information of all mask candidate regions.

[0038] Further, step (6) includes the following steps:

[0039] (61) The obtained activation mask candidate regions are combined into a set m f Fuse it with the visual features obtained in step (2) to obtain the current mask candidate region m f The visual features of the foreground object and the feature vector of the entire image are connected to obtain the visual features V used to reconstruct the query re :

[0040]

[0041] (62) Use the convolutional layer to reduce the dimension of the current visual feature and obtain a new query reconstruction visual feature V' re , using the decoder RNN (i.e. RNN re ) to incrementally predict each word in the query:

[0042] f t =RNN re (V' re ,W e q t-1 ,h t-1)

[0043]

[0044] where q t is the tth word predicted in the query, W e is the word embedding matrix, h t-1 Representing RNN re The hidden state output at step (t-1), f t is the feature of the tth word in the query, softmax represents softmax normalization; RNN re According to the visual feature vector V' re and features of previous words in the query to predict the tth word in the query;

[0045] (63) Calculate the Euclidean distance between the reconstructed query and the input query as the query consistency loss:

[0046]

[0047] Where x is the feature vector of the current query; ht represents the feature vector of the reconstructed query, ||h t -x||2 is h t and x; Loss(Q,Q') is the query reconstruction consistency loss.

[0048] Furthermore, step (7) is as follows: determine whether the current mask set meets the text requirements. If not, iteratively adjust the attention weight of the cross-modal attention mechanism part and the selection of the activation mask by the "activation gate" part, and calculate the loss again. This iterative cycle is repeated until the selected mask set meets the text requirements and the mask set is output, or the number of iterations exceeds the threshold and an empty set is output, indicating that there is no target in the image that meets the text requirements.

[0049] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements any one of the multi-target image segmentation methods based on cross-modal deep learning.

[0050] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, it implements any one of the multi-target image segmentation methods based on cross-modal deep learning.

[0051] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention uses an unsupervised mask generator to generate segmentation masks of targets in an image, providing an operational prerequisite for multi-target segmentation, and does not require data annotation, thereby reducing time and labor costs; the present invention uses a typical convolutional neural network model and a recursive neural network model to extract features from image and text data, and the model is mature and effective; the "activation gate" proposed in the present invention is essentially a column vector whose elements are either zero or one, which can effectively realize the selective activation of the mask; the present invention uses a query reconstruction loss training model to iteratively select an activation mask, providing a guarantee for the correctness of the output result, and the model is simple as a whole, with low training time and space overhead, and can obtain relatively accurate multi-target segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of the process of the present invention.

[0053] Figure 2 This is the multi-target segmentation model diagram of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, an embodiment of the present invention provides a multi-target image segmentation method based on cross-modal deep learning, comprising the following steps:

[0056] Step 1, denoising and resizing the obtained cucumber image data to obtain an image data set, obtaining text data, using an encoder in DeepLab v3 based on a convolutional neural network model to extract features of an input image I, and using a recurrent neural network model long short-term memory network LSTM to extract features of an input text query Q; comprising the following steps:

[0057] Step 1-1, using a camera to collect cucumber image data, step 1-2, preprocessing the image data, denoising, adjusting to a uniform size, obtaining an image data set, and dividing the data set into a training set, a validation set, and a test set;

[0058] Steps 1-3 use the encoder in DeepLab v3 based on the convolutional neural network model to extract features from the input image I, and extract the features of the input image I as visual features:

[0059] V=CNN(I)={v1, v2,...,v hw}

[0060] Where V∈R h×w×Cv is the feature map of the image, h and w are the height and width of the feature map respectively. In the feature map V, each feature vector vi ∈R Cv Represents the visual features of the i-th region in image I;

[0061] Steps 1-4 use a typical recursive neural network model, namely the long short-term memory network LSTM, to encode the features of the input text query Q:

[0062] x=RNN(W e Q)

[0063] In the formula is a word embedding matrix, For the feature vector of the entire language query Q, the hidden state of the last step that encodes the entire query information is used as the query feature vector.

[0064] Step 2: Select the unsupervised mask generator FreeSOLO to obtain the mask candidate regions of all targets in the input image I in step 1, and use the element-by-element multiplication method to fuse the two-dimensional binary matrix corresponding to the mask with the visual features after dimensionality reduction to obtain the visual features of each mask; the specific process of mask feature acquisition includes:

[0065] Step 2-1, select FreeSOLO to obtain a set of masks m of the target in the input image I, m∈M(I), that is, a set of two-dimensional binarization matrices with the same size as the input image;

[0066] Step 2-2, downsample the target mask from 224×224 to a spatial size that matches the visual feature map;

[0067] Step 2-3, expand the binary matrix corresponding to each mask and multiply it element by element with the visual feature to obtain the visual feature corresponding to each mask:

[0068] V mi =V'⊙m'

[0069] Where V mi Represents the visual feature corresponding to the mask, i∈{1, 2, …, n}, n represents the total number of masks. ⊙ represents the Hadamard product, also known as the element-wise product.

[0070] Step 3: adjust the dimensions of the masked visual features in step 2 and the text features in step 1, use the cross-modal attention mechanism to calculate the similarity between the image sequence and the text sequence to obtain the attention weight, and then perform weighted calculation on the visual feature vector to output the masked visual features fused with the text features; the specific process of cross-modal feature fusion includes:

[0071] Step 3-1, use two linear layers to map visual and text features to the same dimension;

[0072] Step 3-2: The cross-modal attention mechanism is used to capture the correlation between visual features and text features, and feature fusion is performed through the calculation of query, key, and value;

[0073] Step 3-3, encode the text features as Query, encode the visual features as Key and Value, and use the dot product to calculate the similarity between the text and visual features;

[0074] Step 3-4, normalize the similarity of all positions through the softmax function to generate attention weights, which represent the importance of text features to each position in the visual features. Use the attention weights to weight the sum of the visual features to obtain the visual features associated with the text:

[0075]

[0076] Among them, softmax represents the activation function, Q, K, and V represent the query, key, and value encoded by visual and text features, respectively. k The dimension of Embedding.

[0077] Step 4: Set the "activation gate" after the fusion mask feature output by the attention mechanism. The "activation gate" is an n-dimensional column vector, in which each element is either 0 or 1, and the number of elements is the same as the number of candidate mask regions in step 2. Use the filter to filter out the activated candidate mask regions and use the aggregator to aggregate them; the "activation gate" activates the mask. The specific process includes:

[0078] Step 4-1, based on the visual features output by the attention mechanism, use the "activation gate" to select the activated mask. The "activation gate" is essentially a 1×n column vector. When the corresponding element is 1, it indicates activation, and when the corresponding element is 0, it indicates inactivation;

[0079] Step 4-2, using a filter to filter the activation mask to the aggregator layer, using a union operation to achieve the purpose of retaining the area covered by all mask schemes, and performing a bitwise logical "OR" (OR) operation on each mask area;

[0080] Step 4-3, for each pixel position, if the value of any mask scheme at that position is 1 (or a valid value), the final result is 1. The formula examples of the corresponding operations are as follows:

[0081] FinalMask(i,j)=Mask1(i,j)∨Mask2(i,j)∨……∨Mask m (i,j)

[0082] Where (i, j) represents the pixel position, m is the total number of activated mask candidate regions, and finally the union m of all mask candidate regions will be obtained. f , retaining the information of all mask candidate regions.

[0083] Step 5: The mask obtained by aggregation in step 4 is fused with the visual features extracted in step 1, and input into the long short-term memory network for text reconstruction. The Euclidean distance between the reconstructed text features and the input text features is calculated as the consistency loss. The text query reconstruction process includes:

[0084] Step 5-1, activate the mask and set m f Fusion with the obtained visual features to obtain the current mask candidate region m f The visual features of the foreground object and the image are concatenated to obtain the visual features V used to reconstruct the query re :

[0085]

[0086] Step 5-2: Use the convolutional layer to reduce the dimension of the current visual feature and obtain the new query reconstruction visual feature V' re , using the decoder RNN (i.e. RNN re ) to incrementally predict each word in the query:

[0087] f t =RNN re (V' re ,W e q t-1 ,h t-1 )

[0088]

[0089] where q t is the tth word predicted in the query, f t is the feature of the tth word in the query, and softmax is the softmax normalization. RNN re According to the visual feature vector V' re Predict the tth word in the query using features of previous words in the query;

[0090] Step 5-3, RNN re The hidden state ht in the last step contains the information of the entire reconstructed query, which is used as the feature vector of the reconstructed query Q' to compare with the feature vector x of the input text query Q, and the Euclidean distance between the reconstructed query and the input query is calculated as the query consistency loss:

[0091]

[0092] Where x is the feature vector of the current query; ht represents the feature vector of the reconstructed query, ||h t -x||2 is h t The Euclidean distance between q and x. Loss(Q,Q') is the query reconstruction consistency loss.

[0093] Step 6, by reducing the consistency loss in step 5, iteratively adjust the attention weight of the cross-modal fusion part and the activation selection of the mask candidate area by the activation gate part until the consistency loss is lower than the threshold or the number of iterations reaches the preset maximum value. At this time, the activated mask is the segmentation result. Reconstruct the output of the loss supervision model. The specific process includes:

[0094] Through the query reconstruction consistency loss in step 5, determine whether the current mask set meets the text requirements. If not, iteratively adjust the attention weight of the cross-modal attention mechanism part and the selection of the activation mask in the "activation gate" part, and calculate the loss again. This iterative cycle is repeated until the selected mask set meets the text requirements and the mask set is output, or the number of iterations exceeds the threshold and an empty set is output, indicating that there is no target in the image that meets the text requirements.

[0095] An embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the computer program implements any one of the multi-target image segmentation methods based on cross-modal deep learning.

[0096] An embodiment of the present invention further provides a storage medium storing a computer program, which, when executed by a processor, implements any one of the multi-target image segmentation methods based on cross-modal deep learning.

Claims

1. A multi-objective image segmentation method based on cross-modal deep learning, characterized in that: The following steps are involved: (1) Collecting cucumber image data including color, shape, length, thickness and text data; and performing preprocessing; (2) The encoder in DeepLab v3 based on the convolutional neural network model is used to extract the features of the input image, and the recurrent neural network model long short-term memory network LSTM is used to extract the features of the input text; (3) The unsupervised mask generator FreeSOLO is selected to obtain the mask candidate regions of all targets in the input image, and the two-dimensional binary matrix corresponding to the mask is fused with the visual features after dimensionality reduction by element-by-element multiplication to obtain the visual features of each mask; (4) Processing the masked visual features and the text features in step (2) to obtain the corresponding feature sequence, using the cross-modal attention mechanism to calculate the similarity between the image sequence and the text sequence to obtain the attention weight, performing weighted calculation on the visual feature vector, and outputting the masked visual features that incorporate the text features; (5) Setting an "activation gate" after the fused mask features output by the attention mechanism; using filters to filter out activated mask candidate regions and using an aggregator to aggregate them; (6) The mask obtained by aggregation is fused with the visual features extracted in step (2), and input into the long short-term memory network for text reconstruction. The Euclidean distance between the reconstructed text features and the input text features is calculated as the consistency loss; (7) Adjust the attention weight of the cross-modal fusion part and the activation gate part for the activation selection of the mask candidate area until the consistency loss is lower than the threshold or the number of iterations reaches the preset maximum value.

2. According to claim 1, a multi-objective image segmentation method based on cross-modal deep learning is characterized in that: In step (1), the preprocessing includes: denoising, adjusting the image data to a uniform size, and dividing the data set into a training set, a validation set, and a test set.

3. The multi-objective image segmentation method based on cross-modal deep learning according to claim 1, characterized in that: In step (2), feature extraction is performed on the input image to extract the features of the input image as visual features. The formula is as follows: V=CNN(I)={v1,v2,……,v hw } Where V∈R h×w×Cv is the feature map of the image, h and w are the height and width of the feature map respectively. In the feature map V, each feature vector v i ∈R Cv Represents the visual features of the i-th region in image I; The long short-term memory network LSTM is used to encode the features of the input text query Q. The formula is as follows: x=RNN(W e Q) in, is a word embedding matrix, For the feature vector of the entire language query Q, the hidden state of the last step that encodes the entire query information is used as the query feature vector.

4. The multi-objective image segmentation method based on cross-modal deep learning according to claim 1, characterized in that: Step (3) comprises the following steps: (31) FreeSOLO is selected to obtain a set of masks m of the target in the input image I, m∈M(I), that is, a set of two-dimensional binarization matrices with the same size as the input image; (32) Downsample the object mask from 224×224 to a spatial size that matches the visual feature map; (33) Expand the binary matrix corresponding to each mask and multiply it element-by-element with the visual feature to obtain the visual feature corresponding to each mask: V mi =V'⊙m' Among them, V mi Represents the visual feature corresponding to the mask, i∈{1, 2, …, n}, n represents the total number of masks. ⊙ represents the Hadamard product, also known as the element-wise product.

5. The multi-objective image segmentation method based on cross-modal deep learning according to claim 1, characterized in that: Step (4) comprises the following steps: (41) Use two linear layers to map visual and textual features to the same dimension; (42) The cross-modal attention mechanism is used to capture the correlation between visual features and text features, and perform feature fusion through the calculation of query, key, and value; (43) Encode text features as Query, encode visual features as Key and Value, and use dot product to calculate the similarity between text and visual features; (44) The similarities of all positions are normalized by the softmax function to generate attention weights. The visual features are weighted and summed using the attention weights to obtain the visual features associated with the text. The formula is as follows: Among them, softmax represents the activation function, Q, K, and V represent the query, key, and value encoded by visual and text features, respectively. k The dimension of Embedding.

6. The multi-objective image segmentation method based on cross-modal deep learning according to claim 1, characterized in that: In step (5), "activation gate" is an n-dimensional column vector, where each element is either 0 or 1, and the number of elements is the same as the number of mask candidate regions in step (3), including the following steps: (51) Based on the visual features output by the attention mechanism, the activation mask is selected using an “activation gate”. The “activation gate” is essentially a 1×n column vector. When the corresponding element is 1, it indicates activation, and when the corresponding element is 0, it indicates inactivation. (52) Using a filter to filter the activation mask to the aggregator layer, using a union operation to achieve the purpose of retaining the area covered by all mask schemes, and performing a bitwise logical "OR" operation on each mask area; (53) For each pixel position, if the value of any mask scheme at that position is 1 or a valid value, the final result is 1; the formula is as follows: FinalMask(i,j)=Mask1(i,j)∧Mask2(i,j)∨……∨Mask m (i,j) Where (i, j) represents the pixel position, m is the total number of activated mask candidate regions, and finally the union m of all mask candidate regions will be obtained. f , retaining the information of all mask candidate regions.

7. The multi-objective image segmentation method based on cross-modal deep learning according to claim 1, characterized in that: Step (6) comprises the following steps: (61) The obtained activation mask candidate regions are combined into a set m f Fuse it with the visual features obtained in step (2) to obtain the current mask candidate region m f The visual features of the foreground object and the feature vector of the entire image are connected to obtain the visual features V used to reconstruct the query re : (62) Use the convolutional layer to reduce the dimension of the current visual feature and obtain a new query reconstruction visual feature V' re , using the decoder RNN (i.e. RNN re ) to incrementally predict each word in the query: f t =RNN re (V' re ,W e q t-1 ,h t-1 ) where q t is the tth word predicted in the query, W e is the word embedding matrix, h t-1 Representing RNN re The hidden state output at step (t-1), f t is the feature of the tth word in the query, softmax represents softmax normalization; RNN re According to the visual feature vector V' re and features of previous words in the query to predict the tth word in the query; (63) Calculate the Euclidean distance between the reconstructed query and the input query as the query consistency loss: Where x is the feature vector of the current query; ht represents the feature vector of the reconstructed query, ||h t -x||2 is h t and x; Loss(Q,Q') is the query reconstruction consistency loss.

8. The multi-objective image segmentation method based on cross-modal deep learning according to claim 1, characterized in that: Step (7) is as follows: determine whether the current mask set meets the text requirements. If not, iteratively adjust the attention weight of the cross-modal attention mechanism and the selection of the activation mask by the "activation gate" part, and calculate the loss again. This iterative cycle is repeated until the selected mask set meets the text requirements and the mask set is output, or the number of iterations exceeds the threshold and an empty set is output, indicating that there is no target in the image that meets the text requirements.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, a multi-objective image segmentation method based on cross-modal deep learning is implemented according to any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a multi-objective image segmentation method based on cross-modal deep learning is implemented according to any one of claims 1 to 7.

Images