A multi-target image segmentation method based on cross-modal deep learning

Abstract

The application discloses a kind of multi-target image segmentation methods based on cross-modal deep learning, comprising the following steps: (1) acquisition cucumber image data and text data and pre-processing;(2) utilize the encoder in DeepLab v3 based on convolutional neural network model to extract the features of input image, utilize recurrent neural network model long short-term memory network LSTM to extract input text features;(3) select unsupervised mask generator FreeSOLO to obtain the mask candidate region of all targets in input image, and adopt element-by-element multiplication method, obtain the visual features of each mask;(4) process mask visual features and text features, obtain corresponding feature sequence, adopt cross-modal attention mechanism to calculate the similarity between image sequence and text sequence to obtain attention weight;(5) set "activation gate" after the fusion mask feature output by attention mechanism;Utilize filter to screen out activated mask candidate region and adopt aggregator to aggregate;(6) the mask obtained by aggregation is fused with the extracted visual features, input into long short-term memory network to reconstruct text;(7) output result;The application can obtain more accurate multi-target segmentation result.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image segmentation, and particularly relates to a multi-target image segmentation method based on cross-modal deep learning. BACKGROUND

[0002] The multi-target image segmentation task mainly involves the following four aspects of technology: first, image feature extraction is one of the basic tasks of image segmentation, aiming to extract features from the input image that are helpful for segmentation. In current deep learning methods, convolutional neural networks, atrous convolution, visual Transformer, etc. can effectively obtain image information of different scales through the above methods. It is outstanding in image feature extraction. Second, text feature extraction also occupies an important position in the multi-modal segmentation task, especially in the scene combined with natural language description. Word embedding technology, BERT and Transformer architecture, CLIP model and other technologies are widely used in text feature extraction, which can effectively capture context information to help the segmentation model accurately identify and distinguish different target objects. Third, multi-modal feature fusion is a key step in the multi-target reference image segmentation task, and feature fusion through attention mechanism, contrast learning is a relatively mature and effective method. Fourth, text reconstruction using visual features is a powerful guarantee for multi-target reference image segmentation, and the classic recurrent neural network model LSTM is currently widely used. The current task related to the following problems: image data background complexity, cucumber target shape diversity; poor segmentation performance, false segmentation or low segmentation accuracy; high computational complexity of the task, unable to meet the real-time processing requirements. SUMMARY

[0003] The purpose of the application is to provide a multi-target image segmentation method based on cross-modal deep learning, which solves the problem of poor image segmentation performance, false segmentation or low segmentation accuracy.

[0004] Technical scheme: the multi-target image segmentation method based on cross-modal deep learning comprises the following steps:

[0005] (1) collect cucumber image data including color, shape, length, thickness and text data; and perform pretreatment;

[0006] (2) extract the features of the input image using the encoder in DeepLab v3 based on the convolutional neural network model, and extract the input text features using the long short-term memory network LSTM of the recurrent neural network model;

[0007] (3) Select the unsupervised mask generator FreeSOLO to obtain the mask candidate region of all targets in the input image, and use the element-by-element multiplication method to fuse the two-dimensional binary matrix corresponding to the mask with the reduced visual features to obtain the visual features of each mask;

[0008] (4) Process the mask visual features and the text features in step (2) to obtain the corresponding feature sequence, calculate the similarity between the image sequence and the text sequence using the cross-modal attention mechanism to obtain the attention weight, and weight the visual feature vector to output the mask visual features fused with the text features;

[0009] (5) Set an "activation gate" after the fusion mask feature output by the attention mechanism; use a filter to select the activated mask candidate region and use an aggregator to aggregate;

[0010] (6) Fuse the aggregated mask with the visual features extracted in step (2) and input them into the long short-term memory network for text reconstruction, and calculate the Euclidean distance between the reconstructed text features and the input text features as the consistency loss;

[0011] (7) Adjust the attention weight of the cross-modal fusion part and the activation selection of the mask candidate region by the activation gate part until the consistency loss is lower than the threshold or the number of iterations reaches the preset maximum value.

[0012] Further, in step (1), the preprocessing includes: denoising, adjusting the image data to a uniform size, dividing the data set into a training set, a validation set and a test set.

[0013] Further, in step (2), the features of the input image are extracted as visual features, and the formula is as follows:

[0014] V = CNN(I) = {v1, v2, …, vn} hw}

[0015] Where V ∈ R h×w×Cv is the feature map of the image, h and w are the height and width of the feature map respectively, and each feature vector v i ∈ R Cv represents the visual features of the i-th region in the image I;

[0016] The features of the input text query Q are encoded using the long short-term memory network LSTM, and the formula is as follows:

[0017] x = RNN(W e Q)

[0018] Where, is a word embedding matrix, For the feature vector of the entire language query Q, the hidden state of the last step encoding the entire query information is taken as the query feature vector.

[0019] Further, step (3) comprises the following steps:

[0020] (31) Select a set of masks m of the target in the input image I by FreeSOLO, m∈M(I), that is, a set of two-dimensional binary matrices with the same size as the input image;

[0021] (32) Downsample the target mask from 224x224 to the spatial size matching the visual feature map;

[0022] (33) Expand the binary matrix corresponding to each mask and multiply it element by element with the visual feature to obtain the visual feature corresponding to each mask:

[0023] V mi = V'⊙m'

[0024] Where V mi represents the visual feature corresponding to the mask, i∈{1,2,……,n}, n represents the total number of masks. ⊙ represents Hadamard product, also known as element-wise product.

[0025] Further, step (4) comprises the following steps:

[0026] (41) Map the visual and text features to the same dimension using two linear layers;

[0027] (42) Cross-modal attention mechanism is used to capture the correlation between visual and text features, and feature fusion is performed through the calculation of Query, Key, and Value;

[0028] (43) Encode the text feature as Query and the visual feature as Key and Value, and use dot product to calculate the similarity between the text and the visual feature;

[0029] (44) Normalize the similarity of all positions by the softmax function to generate attention weights, and use the attention weights to weight sum the visual features to obtain the visual features associated with the text, as follows:

[0030]

[0031] Where softmax represents the activation function, Q, K, and V represent Query, Key, and Value encoded by visual and text features, respectively, and d k is the dimension of Embedding.

[0032] Further, in step (5), the "activation gate" is an n-dimensional column vector, where each element is either 0 or 1, and the number of elements is the same as the number of mask candidate regions in step (3), including the following steps:

[0033] (51) Based on the visual features output by the attention mechanism, the activated mask is selected using the "activation gate", which is essentially a 1xn column vector. When the corresponding element is 1, it means activation, and when the corresponding element is 0, it means non-activation;

[0034] (52) Use the filter to filter the activated mask to the aggregator layer, and use the union operation to retain all mask scheme coverage areas. Perform bitwise logical OR operation on each mask region;

[0035] (53) For each pixel position, the value of any mask scheme at that position is 1 or valid, and the final result is 1; The formula is as follows:

[0036] FinalMask(i,j)=Mask1(i,j)∨Mask2(i,j)∨……∨Mask m (i,j)

[0037] Where (i,j) represents the pixel position, and m is the total number of activated mask candidate regions, and the union of all mask candidate regions m f is obtained, retaining the information of all mask candidate regions.

[0038] Further, step (6) includes the following steps:

[0039] (61) Fuse the obtained activated mask candidate region union m f with the visual features obtained in step (2) to obtain the visual features of the current mask candidate region m f , and connect the feature vectors of the foreground object and the entire image to obtain the visual features V re for query reconstruction:

[0040]

[0041] (62) Use the convolution layer to reduce the dimension of the current visual feature to obtain a new query reconstruction visual feature V' re , and use the decoder RNN (i.e. RNN re ) to predict each word in the query step by step:

[0042] f t =RNN re (V' re ,W e q t-1 ,h t-1)

[0043]

[0044] where q t is the t-th word predicted in the query, W e is the word embedding matrix, h t-1 represents the RNN re hidden state output in the (t-1)-th step, f t is the feature of the t-th word in the query, sof tmax represents the softmax normalization; RNN re predicts the t-th word in the query according to the visual feature vector V’ re and the features of previous words in the query.

[0045] (63) Calculate the Euclidean distance between the reconstructed query and the input query as the query consistency loss:

[0046]

[0047] where x is the feature vector of the current query; ht represents the feature vector of the reconstructed query, ||h t -x||2 is the Euclidean distance between h t and x; Loss(Q, Q') is the query reconstruction consistency loss.

[0048] Further, step (7) is specifically as follows: it is judged whether the current mask set meets the text requirement, if not, the attention weight of the cross-modal attention mechanism part and the selection of the activation mask by the "activation gate" part are iteratively adjusted, the loss is calculated again, and the iteration cycle is repeated until the selected mask set meets the text requirement and the mask set is output, or the iteration number exceeds the threshold and the empty set is output, representing that there is no target in the image that meets the text requirement.

[0049] The electronic device provided by the application comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the computer program realizes the multi-target image segmentation method based on cross-modal deep learning when loaded into the processor.

[0050] The storage medium provided by the application stores a computer program, and the computer program realizes the multi-target image segmentation method based on cross-modal deep learning when executed by a processor.

[0051] Beneficial effects: compared with the prior art, the present application has the following remarkable advantages: the present application uses an unsupervised mask generator to generate a segmentation mask of a target in an image, provides an operation premise for multi-target segmentation, and does not require data labeling, thereby reducing time and labor costs; the present application uses a typical convolutional neural network model and a recurrent neural network model to extract features of image and text data, and the model is mature and has good effects; the "activation gate" proposed in the present application is essentially a column vector with each element being zero or one, which can effectively realize selective activation of the mask; the present application uses query reconstruction loss to train the model to iteratively select the activated mask, thereby providing a guarantee for the correctness of the output result, and the model is simple as a whole, has small training time and space costs, and can obtain a relatively accurate multi-target segmentation result. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of the present application.

[0053] Figure 2 is a multi-target segmentation model of the present application. DETAILED DESCRIPTION

[0054] The technical solutions of the present application will be further described below in combination with the drawings.

[0055] As shown in Figure 1 , the embodiment of the present application provides a multi-target image segmentation method based on cross-modal deep learning, comprising the following steps:

[0056] Step 1, pre-process the obtained cucumber image data to obtain an image data set by denoising and size adjustment, obtain text data, extract features of the input image I using an encoder in DeepLab v3 based on a convolutional neural network model, and extract features of the input text query Q using a long short-term memory network LSTM of a recurrent neural network model; comprising the following steps:

[0057] Step 1-1, use a camera to collect cucumber image data, step 1-2, pre-process the image data by denoising and adjusting to a uniform size to obtain an image data set, and divide the data set into a training set, a validation set and a test set;

[0058] Step 1-3, extract features of the input image I using an encoder in DeepLab v3 based on a convolutional neural network model, and extract features of the input image I as visual features:

[0059] V=CNN(I)={v1,v2,…,v hw}

[0060] where V∈R h×w×Cv is a feature map of the image, h and w are the height and width of the feature map respectively, and each feature vector v in the feature map Vi ∈R Cv denotes the visual feature of the i-th region in the image I;

[0061] Step 1-4, a typical recurrent neural network model, i.e., long short-term memory network LSTM, is used to encode the features of the input text query Q:

[0062] x = RNN(W e Q)

[0063] wherein is a word embedding matrix, is a feature vector of the entire language query Q, and the hidden state of the last step encoding the entire query information is taken as the query feature vector.

[0064] Step 2, select the unsupervised mask generator FreeSOLO to obtain all mask candidate regions of the target in the input image I in step 1, and use the element-wise multiplication method to fuse the two-dimensional binary matrix corresponding to the mask with the reduced visual features to obtain the visual features of each mask; mask feature acquisition, the specific process includes:

[0065] Step 2-1, select FreeSOLO to obtain a set of masks m of the target in the input image I, m ∈ M(I), i.e., a set of two-dimensional binary matrices with the same size as the input image;

[0066] Step 2-2, downsample the target mask from 224x224 to a spatial size matching the visual feature map;

[0067] Step 2-3, expand the binary matrix corresponding to each mask and perform element-wise multiplication with the visual features to obtain the visual features corresponding to each mask:

[0068] V mi = V' ⊙ m'

[0069] wherein V mi represents the visual feature corresponding to the mask, i ∈ {1, 2, …, n}, n represents the total number of masks, and ⊙ represents Hadamard product, also known as element-wise product.

[0070] Step 3, adjust the dimensions of the mask visual features in step 2 and the text features in step 1, use the cross-modal attention mechanism to calculate the similarity between the image sequence and the text sequence to obtain the attention weight, and then weight the visual feature vector to output the mask visual features fused with the text features; cross-modal feature fusion, the specific process includes:

[0071] Step 3-1, map the visual and text features to the same dimension using two linear layers;

[0072] Step 3-2, the cross-modal attention mechanism is used to capture the correlation between visual features and text features, and feature fusion is performed through the calculation of Query, Key, and Value;

[0073] Step 3-3, the text features are encoded as Query, the visual features are encoded as Key and Value, and the similarity between the text and the visual features is calculated using dot product;

[0074] Step 3-4, the similarity of all positions is normalized by the softmax function to generate attention weights, which represent the importance of each position in the visual features to the text features, and the visual features associated with the text are obtained by weighted summation using the attention weights:

[0075]

[0076] where softmax represents the activation function, Q, K, and V represent Query, Key, and Value encoded by visual and text features, respectively, d k is the dimension of Embedding.

[0077] Step 4, an "activation gate" is set after the fusion mask feature output by the attention mechanism, which is an n-dimensional column vector, where each element is either 0 or 1, and the number of elements is the same as the number of mask candidate regions in step 2. The activated mask candidate regions are filtered out using a filter and aggregated using an aggregator; the "activation gate" activates the mask, and the specific process includes:

[0078] Step 4-1, based on the visual features output by the attention mechanism, the "activation gate" is used to select the activated mask, and the "activation gate" is essentially a 1x n column vector. When the corresponding element is 1, it means activation, and when the corresponding element is 0, it means non-activation;

[0079] Step 4-2, the filter is used to filter the activated mask to the aggregator layer, and the union operation is used to retain all mask scheme coverage areas, and the bitwise logical "or" (OR) operation is performed on each mask region;

[0080] Step 4-3, for each pixel position, the value of any mask scheme at that position is 1 (or valid value), and the final result is 1. An example of the corresponding operation formula is as follows:

[0081] FinalMask(i,j)=Mask1(i,j)∨Mask2(i,j)∨……∨Mask m (i,j)

[0082] where (i, j) represents the pixel position, m is the total number of activated mask candidate regions, and the union of all mask candidate regions m is finally obtained f , and the information of all mask candidate regions is retained.

[0083] Step 5, fuse the mask obtained by aggregation in step 4 and the visual features extracted in step 1, input into the long short-term memory network for text reconstruction, and calculate the Euclidean distance between the reconstructed text features and the input text features as the consistency loss; text query reconstruction, the specific process includes:

[0084] Step 5-1, fuse the activated mask union m f and the obtained visual features to obtain the visual features of the current mask candidate region m f , and connect the feature vectors of the foreground object and the image to obtain the visual features V re used for query reconstruction:

[0085]

[0086] Step 5-2, reduce the dimension of the current visual features using a convolutional layer to obtain new query reconstruction visual features V’ re , and use the decoder RNN (i.e. RNN re ) to predict each word in the query step by step:

[0087] f t = RNN re (V' re , W e q t-1 , h t-1 )

[0088]

[0089] where q t is the t-th word predicted in the query, f t is the feature of the t-th word in the query, and softmax is the softmax normalization. RNN re predicts the t-th word in the query according to the visual feature vector V’ re and the features of the previous words in the query;

[0090] Step 5-3, the hidden state ht of the last step in RNN re contains the information of the entire reconstructed query, which is used as the feature vector of the reconstructed query Q’ and compared with the feature vector x of the input text query Q to calculate the Euclidean distance between the reconstructed query and the input query as the query consistency loss:

[0091]

[0092] wherein x is the feature vector of the current query; ht represents the feature vector of the reconstructed query, and ||h t -x||2 is the Euclidean distance between h t and x. Loss(Q, Q') is the query reconstruction consistency loss.

[0093] Step 6: The attention weight of the cross-modal fusion part and the activation selection of the activation gate part to the mask candidate region are iteratively adjusted by reducing the consistency loss in step 5 until the consistency loss is lower than a threshold or the number of iterations reaches a preset maximum value, at which time the activated mask is the segmentation result. The reconstruction loss supervises the model output, and the specific process includes:

[0094] Through the query reconstruction consistency loss in step 5, it is determined whether the current mask set meets the text requirements. If not, the attention weight of the cross-modal attention mechanism part and the selection of the activated mask by the activation gate part are iteratively adjusted, and the loss is calculated again. The iteration cycle is repeated until the selected mask set meets the text requirements and the mask set is output, or the number of iterations exceeds the threshold and an empty set is output, representing that there is no target in the image that meets the text requirements.

[0095] The embodiment of the present application also provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements any one of the multi-target image segmentation methods based on cross-modal deep learning.

[0096] The embodiment of the present application also provides a storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any one of the multi-target image segmentation methods based on cross-modal deep learning.

Claims

1. A multi-object image segmentation method based on cross-modal deep learning, characterized in that, The method comprises the following steps: (1) collecting cucumber image data including color, shape, length, thickness and text data, and performing preprocessing; (2) extracting features of the input image by using an encoder in DeepLab v3 based on a convolutional neural network model, and extracting input text features by using a recurrent neural network model, i.e., a long short-term memory network (LSTM); (3) selecting a FreeSOLO unsupervised mask generator to obtain mask candidate regions of all targets in the input image, and adopting an element-by-element multiplication method to fuse a two-dimensional binary matrix corresponding to the mask with the reduced visual features to obtain visual features of each mask; (4) processing the mask visual features and the text features in step (2) to obtain corresponding feature sequences, calculating the similarity between the image sequences and the text sequences by using a cross-modal attention mechanism to obtain attention weights, weighting the visual feature vectors, and outputting mask visual features fused with the text features; (5) setting an activation gate after the fusion mask features output by the attention mechanism; filtering out activated mask candidate regions by using a filter and aggregating the activated mask candidate regions by using an aggregator; the activation gate is an n-dimensional column vector, each element of which is either 0 or 1, and the number of elements is the same as the number of mask candidate regions in step (3), comprising the following steps: (51) selecting activated masks by using the activation gate based on the visual features output by the attention mechanism; the activation gate is essentially a 1xn column vector, and when the corresponding element is 1, it indicates activation, and when the corresponding element is 0, it indicates inactivation; (52) filtering the activated masks to the aggregator layer by using the filter, and adopting a union operation to achieve the purpose of retaining all mask scheme coverage areas, and performing a bitwise logical or operation on each mask region; (53) for each pixel position, the value of any mask scheme at the position is 1 or a valid value, and the final result is 1; the formula is as follows: FinalMask(i,j) = Mask1(i,j) V Mask2(i,j) V... V Mask m (i,j) where (i, j) represents the pixel position, m is the total number of activated mask candidate regions, and the union of all mask candidate regions m is finally obtained f , the information of all mask candidate regions is retained; (6) fusing the mask obtained by aggregation with the visual features extracted in step (2), inputting into the long short-term memory network for text reconstruction, and calculating the Euclidean distance between the reconstructed text features and the input text features as the consistency loss; (7) adjusting the attention weight of the cross-modal fusion part and the activation selection of the activation gate part on the mask candidate regions until the consistency loss is lower than a threshold value or the number of iterations reaches a preset maximum value.

2. The multi-object image segmentation method based on cross-modal deep learning according to claim 1, characterized in that, In step (1), the preprocessing includes denoising, adjusting the image data to a uniform size, and dividing the data set into a training set, a validation set and a test set.

3. The multi-object image segmentation method based on cross-modal deep learning according to claim 1, characterized in that, In step (2), the features of the input image are extracted as visual features, and the formula is as follows: V = CNN(I) = {v1, v2,..., vn} where n is the number of output neurons of the CNN. hw} wherein V ∈ R h×w×Cv is a feature map of the image, h and w are the height and width of the feature map, respectively, and each feature vector v i ∈ R Cv represents the visual feature of the i-th region in the image I; The long short-term memory network (LSTM) is used to encode the features of the input text query Q, and the formula is as follows: x = RNN(W e Q) wherein, is a word embedding matrix, is a feature vector for the entire language query Q, the last step of the hidden state that encodes the entire query information is taken as the query feature vector.

4. The multi-object image segmentation method based on cross-modal deep learning according to claim 1, characterized in that, Step (3) comprises the following steps: (31) selecting a FreeSOLO to obtain a set of masks m of the target in the input image I, m∈M(I), i.e., a set of two-dimensional binary matrices with the same size as the input image; (32) down-sampling the target mask from 224x224 to a spatial size matching the visual feature map; (33) Extend the binary matrix corresponding to each mask, and multiply it element by element with the visual features to obtain the visual features corresponding to each mask: V mi = V' ⊙ m' where V mi represent the visual features corresponding to the mask, i ∈ {1, 2, …, n}, n represents the total number of masks; and represents the Hadamard product, also known as the element-wise product.

5. The multi-object image segmentation method based on cross-modal deep learning according to claim 1, characterized in that, Step (4) includes the following steps: (41) Map the visual and text features to the same dimension using two linear layers; (42) Cross-modal attention mechanism is used to capture the correlation between visual and text features, and feature fusion is performed through the calculation of Query, Key, and Value; (43) Encode the text features as Query and the visual features as Key and Value, and use dot product to calculate the similarity between text and visual features; (44) Normalize the similarity of all positions by the softmax function to generate attention weights, and use the attention weights to weight sum the visual features to obtain the visual features associated with the text, as follows: where softmax denotes an activation function, Q, K, V represent Query, Key, Value encoded by visual and text features, respectively, d k is the dimension of Embedding.

6. The multi-object image segmentation method based on cross-modal deep learning according to claim 3, characterized in that, Step (6) includes the following steps: (61) The union of the obtained activation mask candidate regions and m f fusion with the visual features obtained in step (2), to obtain the current mask candidate region m f The visual features of the foreground object and the entire image are connected to obtain the visual features V re for reconstruction query: (62) reduce the dimensionality of the current visual feature using a convolutional layer to obtain a new query reconstructed visual feature V re , using a decoder RNN RE, i.e. RNN re predict each word in the query step by step: f t = RNN re (V re , W e q t-1 , h t-1 ) Where q t W is the predicted t-th word in the query. e For the word embedding matrix, h t-1 RNN re The hidden state output at step t-1, f t For the features of the t-th word in the query, sof tmax represents softmax normalization; RNN re Based on the visual feature vector V' re The features of previous words in the query are used to predict the t-th word in the query; (63) Calculate the Euclidean distance between the reconstructed query and the input query as the query consistency loss: where x is the feature vector of the current query; ht represents the feature vector of the reconstructed query, ||h t -x||2 is the Euclidean distance between h t and x; and Loss(Q, Q') is the query reconstruction consistency loss.

7. The multi-object image segmentation method based on cross-modal deep learning according to claim 1, characterized in that, Step (7) is as follows: judge whether the current mask set meets the text requirements, if not, iteratively adjust the attention weights of the cross-modal attention mechanism part and the selection of the activation mask by the activation gate part, and calculate the loss again, and the iteration is repeated until the selected mask set meets the text requirements and the mask set is output, or the iteration number exceeds the threshold and the empty set is output, representing that there is no target in the image that meets the text requirements.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program, when loaded into the processor, implements a multi-target image segmentation method based on cross-modal deep learning according to any one of claims 1-7.

9. A storage medium storing a computer program, characterized by The computer program, when executed by the processor, implements a multi-target image segmentation method based on cross-modal deep learning according to any one of claims 1-7.

Images