A referential segmentation method based on selective fusion of multi-scale features

Abstract

The application discloses a kind of based on multi-scale feature selection fusion's reference segmentation method, considering in the practical application of reference segmentation, the difficult alignment between multimodal features, difficult sufficient interaction and background feature difficult to suppress The problem has not been effectively solved, resulting in inaccurate positioning and model precision reduction problem, the application designs the fusion module based on text guide, shallow to deep layer of decoder is carried out multi-scale feature selection, preliminary selection is carried out the feature that meets text content, only contains target visual feature by secondary selection mechanism Further filtering, and on this basis, we design attention contrast loss function, it can effectively suppress background feature, improve model precision, by post-processing segmentation module, according to image segmentation characteristics, different sampling rate is used to model global modeling, segmentation obtains final result, by benchmark data set verification, the application can greatly improve segmentation precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of end-to-end referring segmentation based on deep learning, and in particular to a referring segmentation method based on selective fusion of multi-scale features. BACKGROUND

[0002] In recent years, referring image segmentation has attracted great interest in the academic community and has been widely studied. The definition of referring image segmentation is: given an image and a text expression consistent with the image scenario, the instance object referred to by the text is segmented from the image. Unlike traditional image segmentation, referring image segmentation is a kind of image segmentation combined with text expression, which is a multi-modal task. Since multi-modal data is more in line with the way humans perceive the world, this task has more extensive engineering significance in real life. For example, in the field of robot perception and control, humans interact with robots through dialogue, and robots segment and grasp the corresponding target object according to the text information and corresponding visual image information of the dialogue to complete the closed loop of human-computer interaction. In the field of image matting, the corresponding target can be conveniently and quickly cut out through a text or voice.

[0003] The previous research methods all use convolutional neural network CNN and recurrent neural network RNN to extract visual features and text features respectively, then concatenate the two features in the channel, and then design a fusion module to extract multi-modal features, and finally send them into the segmentation module for segmentation to obtain the segmentation result of the target. These methods have achieved remarkable results, but two main problems have not been effectively solved.

[0004] First, visual features and text features are two different forms of representation. When fusing two modal data, the alignment problem between modalities needs to be considered to align the corresponding words in the text with the corresponding target visual features in the image to ensure the accuracy of the segmentation result. The difficult alignment problem of multi-modal features has not been effectively solved in previous methods, which makes the interaction between visual features and text features insufficient, resulting in inaccurate positioning of the model and reduced accuracy. Second, the expression of text has high randomness, and the language used by different people to express the same meaning is quite different. In addition, the text may contain words unrelated to the target. According to the processing method of the previous method, these unrelated features cannot be effectively suppressed, and these features will be segmented as foreground into the result, making the result inaccurate. SUMMARY

[0005] In order to solve the problems of difficult alignment, difficult full interaction and difficult inhibition of background features among multi-modal features in the prior art, which cause inaccurate positioning and reduced model precision, the purpose of the present application is to provide a referential segmentation method based on selective fusion of multi-scale features, which aims to greatly improve the segmentation accuracy.

[0006] According to a first aspect of the present application, a referential segmentation method based on selective fusion of multi-scale features is provided, characterized in that the segmentation model comprises a DarkNet53 backbone network (Redmon J, Farhadi A. YOLOv3: An Incremental Improvement [J]. arXiv e-prints, 2018), a bidirectional GRU (bidirectional recurrent neural network Gate Recurrent Unit), a language-guided feature selective jump module, a secondary selection module, and an image post-processing segmentation module. The method comprises:

[0007] Step 10: visual feature extraction, for a given RGB image, first scaling and cutting, and then obtaining three different scale visual features F v1 , F v2 , F v3 through the encoder of DarkNet53.

[0008] Step 20: text feature extraction, for a given text, first segmenting and converting into vector form, then using bidirectional GRU to extract preliminary text features, and finally weighting and summing according to the importance of each word based on the text attention mechanism to obtain the text feature F l .

[0009] Step 30: through the fusion part of the language-guided feature selective jump module, first select F v1 , F v2 , F v3 to obtain the visual feature related only to the text description, and then through the jump part of the language-guided feature selective jump module to jump connect the selected features.

[0010] Step 30 comprises steps 31 to 34.

[0011] Step 31: input F v1 , F v2 , F v3 into the fusion part of the language-guided feature selective jump module, and perform dot product operation with F l to obtain the feature after dot product fusion , wherein W l represents stacking F l to the same width and height as the visual feature, denotes a 1x1 convolutional layer operation, denotes a matrix point multiplication operation, and g denotes a Leaky-Relu activation function.

[0012] Step 32: inputting the spatial attention map obtained in step 31 into the spatial fusion branch to obtain a spatial attention feature map inputting the channel attention map obtained in step 32 into the channel fusion branch to obtain a channel attention feature map W v , W q , and W z denote a 1x1 convolutional layer operation, F SG denotes a Sigmoid operation, W q denotes a Softmax operation, F GP denotes a global pooling operation.

[0013] Step 33: multiplying the attention map obtained in step 32 with the visual feature to obtain a visual feature selected by channel attention and a visual feature selected by spatial attention to obtain a visual feature related only to the text description

[0014] Step 34: performing jump connection by the jump part of the language-guided feature selection and jump module to obtain a multi-scale fused visual feature wherein the feature after upsampling of the visual feature of one layer concat denotes concatenation of features in the channel, denotes convolution operation on the feature , Up denotes upsampling operation.

[0015] Step 40: screening and positioning the target feature by the secondary selection module to obtain a visual feature containing only the target wherein the fusion feature after dot product operation of the visual feature and the text feature wherein W l ' denotes stacking F l to the same width and height as the visual feature, W' v denotes a 1x1 convolutional layer operation.

[0016] Step 50: obtaining a segmentation prediction value by the image post-processing segmentation module.​​​​​

[0017] Step 60: training the bidirectional GRU in the segmentation model, the language dominant-based feature selection jump module, the secondary selection module and the image post-processing segmentation module using the training data, wherein the segmentation prediction loss used for training is , the attention contrast loss used for training is y i , P n represents the positive sample in y i , and the total loss function used for training is L = aL seg + bL acl , a and b are hyperparameters, and a and b represent the proportion of the segmentation prediction loss and the attention contrast loss, respectively.

[0018] Further, the reference segmentation method based on the selective fusion of multi-scale features provided by the application is characterized in that step 10 further comprises: step 11: cutting the given RGB image into a size of 416*416 and then inputting the image into the DarkNet53 backbone network; and step 12: extracting the visual features F v1 , the visual feature F v2 of the 152th layer and the visual feature F v3 of the 69th layer.

[0019] The DarkNet53 backbone network comprises a CBL module and a ResX module, wherein the CBL module comprises a convolution layer, a data standardization layer and a Leaky-Relu activation layer, and the ResX module comprises a Res1 module, a Res2 module, two Res8 modules and a Res4 module.

[0020] Further, the reference segmentation method based on the selective fusion of multi-scale features provided by the application is characterized in that step 20 further comprises: step 21: performing word segmentation on the given text to obtain T [w1, w2, …, w n ], n represents the length of the text; step 22: according to the GloVe model (Global Vectors for Word Representation model), the segmented vectors are converted into a text vector with a length of n, and the text vector is input into the bidirectional GRU network to obtain the preliminary text feature t [1, 2, 3, …, n], wherein, represents the forward GRU operation, represents the reverse GRU operation; and step 23: obtaining the text feature

[0021] Further, the reference segmentation method based on selective fusion of multi-scale features provided by the application is characterized in that step 50 further comprises:

[0022] Step 51: F twice The first to fifth parallel operations are performed in parallel, wherein: the first parallel operation is a 1*1 convolution operation; the second parallel operation is a 3*3 convolution operation with a hole convolution rate of 6 and a padding of 6; the third parallel operation is a 3*3 convolution operation with a hole convolution rate of 12 and a padding of 12; the fourth parallel operation is a 3*3 convolution with a hole convolution rate of 18 and a padding of 18; and the fifth parallel operation is a pooling operation, which comprises a global average pooling operation, a 1*1 convolution operation and an up-sampling operation to restore the size of the input.

[0023] Step 52: The first to fifth parallel operation results are spliced in the channel, and then a 1*1 convolution operation is performed to reduce the dimension.

[0024] Step 53: The reduced dimension result is iteratively subjected to a twice up-sampling operation and then a convolution operation until the size is equivalent to that of the real segmentation graph, so as to obtain a segmentation prediction result.

[0025] According to the second aspect of the application, a computer device is provided, characterized by comprising: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the reference segmentation method based on selective fusion of multi-scale features of the first aspect.

[0026] According to the third aspect of the application, a computer readable storage medium is provided, characterized by storing instructions, which are executed by a processor to execute the reference segmentation method based on selective fusion of multi-scale features of the first aspect.

[0027] Compared with the prior art, the above technical solutions conceived by the application have at least the following beneficial effects:

[0028] 1. A language dominant feature fusion module is designed, which fuses visual features and text features on both channel and spatial levels, so that sufficient information interaction between modalities is realized;

[0029] 2. Based on the above fusion module, a secondary selection mechanism and an attention contrast loss function are proposed, which strengthen the guidance of text to visual features and can accurately locate the position of the target in the image. This suppresses irrelevant background features and improves the accuracy of the model;

[0030] 3. A reference segmentation framework based on multi-scale selective fusion is designed, and the design idea is twice selection. The first selection is to select visual features related to the text, and the second selection is to select visual features containing only the target. Through this step-by-step selection idea, the difficulty of direct segmentation of multi-modal features is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0032] Figure 1 is a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment.

[0033] Figure 2 is a reference segmentation method based on multi-scale feature selective fusion compared with prior art according to an example embodiment.

[0034] Figure 3 is a network structure diagram of a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment.

[0035] Figure 4 is a language dominant feature fusion module diagram of a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment.

[0036] Figure 5 is an ablation experiment diagram of a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment.

[0037] Figure 6 is a segmentation result diagram of a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment.

[0038] Figure 7 is an IoU result comparison diagram of a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment.

[0039] Figure 8 is a Prec@X result comparison diagram of a reference segmentation method based on multi-scale feature selective fusion according to an example embodiment. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0041] The present application relates to a referential segmentation method based on multi-scale feature selective fusion, characterized in that the segmentation model MDSFNet of the present application includes a DarkNet53 backbone network (Redmon J, Farhadi A. YOLOv3: An Incremental Improvement [J]. arXiv e-prints, 2018), a bidirectional GRU (bidirectional recurrent neural network Gate Recurrent Unit), a language-guided feature selective jump module, a secondary selection module, and an image post-processing segmentation module.

[0042] As shown in Figure 1 , the method comprises steps 10 to 50.

[0043] Step 10: visual feature extraction, for a given RGB image, first scaling and cutting, and obtaining three different scale visual features F v1 , F v2 , and F v3 through the encoder of DarkNet53.

[0044] This step uses DarkNet53 as the backbone network to extract rich multi-scale visual features.

[0045] In some embodiments, step 10 further comprises: step 11: cutting the given RGB image to a size of 416x416 and inputting it into the DarkNet53 backbone network; step 12: extracting the visual features F v1 , the visual features F v2 of the 152th layer, and the visual features F v3 of the 69th layer.

[0046] The Darknet53 backbone network includes CBL modules and ResX modules, wherein the CBL module includes a convolution layer, a data standardization layer and a Leaky-Relu activation layer, and the ResX module includes a Res1 module, a Res2 module, two Res8 modules and a Res4 module.

[0047] Step 20: text feature extraction, for a given text, first, the text is segmented and converted into a vector form, then the preliminary text features are extracted using bidirectional GRU, and finally, the text features F are obtained by weighted summation based on the text attention mechanism according to the importance of each word l .

[0048] In some embodiments, step 20 further comprises: step 21: segmenting the given text to obtain T∈[w1,w2,…,wn], where n represents the length of the text; step 22: according to the GloVe model (Global Vectors for Word Representation model), the segmented vector is obtained, and the text vector with a length of n is obtained, and the text vector is input into the bidirectional GRU network to obtain the preliminary text feature n t∈[1,2,3,...,n], wherein, represents the forward GRU operation, represents the reverse GRU operation; step 23: obtaining the text feature F according to the text attention mechanism We input h t into the text attention mechanism, so that each word gets different attention, and the weight of the word that meets the segmentation target is larger, so as to reduce the influence of irrelevant words on segmentation.

[0049] Step 30: through the fusion part of the language dominant feature selective jump module, F v1 , F v2 , F v3 are selected to obtain the visual features related only to the text description, and then the selected features are connected through the jump part of the language dominant feature selective jump module.

[0050] The three visual features of step 10 contain a large number of rich image features, which are selected by the designed language dominant feature fusion module, so that the text features and the visual features can be fully fused, and the visual features related only to the text description are obtained, and then they are connected to the decoder part of the model.

[0051] Specifically, step 30 comprises steps 31 to 34.

[0052] Step 31: input F v1 , F v2 , F v3 into the fusion part of the language dominant feature selective jump module, and perform dot product operation with F l to obtain the feature after dot product fusion ​This dot product operation is similar to doing a preliminary similarity measurement, which can obtain more detailed information.

[0053] where W l represents the F l stacked to the same width and height as the visual features, represents a 1x1 convolutional layer operation, represents a matrix dot product operation, and g represents a Leaky-Relu activation function.

[0054] The fusion part has two branches, one branch focuses on space and one branch focuses on channels, and both branches will finally generate an attention map.

[0055] Step 32: input to the spatial fusion branch to obtain a spatial attention map input to the channel fusion branch to obtain a channel attention map W v , W q , and W z represent a 1x1 convolutional layer operation, F SG represents a Sigmoid operation, W q represents a Softmax operation, and F GP represents a global pooling operation.

[0056] Step 33: multiply the attention map with the visual features to obtain visual features selected by channel attention and visual features selected by spatial attention to obtain visual features related only to the text description

[0057] Step 34: jump through the jump part of the language-guided feature selection jump module to connect and splice and fuse with the features after upsampling of the visual features of the previous layer to obtain multi-scale fused visual features where the features after upsampling of the visual features of one layer concat represents spliced features in the channel, represents convolution operation on , represents convolution operation on , and Up represents upsampling operation.

[0058] Step 40: screen and locate the target features through the secondary selection module to obtain visual features containing only the target​ wherein the fusion feature after the dot product operation of the visual feature and the text feature is again wherein W′ l represents performing F l stacked to the same width and height as the visual feature, W′ v represents a 1x1 convolutional layer operation.

[0059] In step 30, we extract visual features with rich semantic information from shallow to deep multi-scale, which are only related to text features. In order to select visual features containing only the target, we need to perform secondary selection. And in order to suppress irrelevant background features, we design an attention contrast loss for training in step 60, so that the segmentation accuracy is higher.

[0060] Step 50: obtaining a segmentation prediction value through an image post-processing segmentation module.

[0061] Since the visual feature is very deep, the deeper the depth, the more details lost by the CNN convolution operation. Therefore, we use multi-scale feature extraction and atrous convolution to obtain rich pixel information and expand the receptive field, so that the model segmentation effect is better.

[0062] Specifically, in some embodiments, step 50 further comprises steps 51 to 53.

[0063] Step 51: performing a 1x1 convolution operation on F twice The first to fifth parallel operations are performed in parallel, wherein: the first parallel operation is a 1x1 convolution operation; the second parallel operation is a 3x3 convolution operation with an atrous convolution rate of 6 and a padding of 6; the third parallel operation is a 3x3 convolution operation with an atrous convolution rate of 12 and a padding of 12; the fourth parallel operation is a 3x3 convolution with an atrous convolution rate of 18 and a padding of 18; and the fifth parallel operation is a pooling operation, followed by a 1x1 convolution operation and finally an up-sampling operation to restore the input size.

[0064] Step 52: splicing the first to fifth parallel operation results in the channel, and finally performing a 1x1 convolution operation to reduce the dimension.

[0065] Step 53: iteratively performing an up-sampling operation twice and then a convolution operation on the reduced dimension result until the size is comparable to that of the real segmentation map to obtain a segmentation prediction result.

[0066] Step 60: training the bidirectional GRU, the language-based feature selective jump module, the secondary selection module and the image post-processing segmentation module in the segmentation model using training data, wherein the segmentation prediction loss used for training is represents the predicted value, and the attention contrast loss used for training is y i denotes the real segmentation map, P n denotes y i positive samples in y, the total loss function used in training is L = aL seg + bL acl , a and b are hyperparameters, denoting the proportion of segmentation prediction loss and attention contrast loss, respectively.

[0067] Step 60 includes designing an attention contrast loss function for the secondary selection mechanism. The attention contrast loss function can perform contrastive learning on positive and negative samples, so that positive samples are close to positive samples, negative samples are close to negative samples, and positive samples are far away from negative samples, so as to suppress irrelevant background features and improve the segmentation effect of the model

[0068] We verified the detection accuracy of the referential segmentation method based on multi-scale feature selective fusion provided by the application, as follows:

[0069] We tested our results on three benchmark datasets, RefCOCO, RefCOCO+, and G-Ref. Among them, the RefCOCO dataset contains 19,994 images, including 50,000 objects and 142,210 corresponding text expressions. The images and text expressions are collected from MSCOCO and a two-player interactive game, with an average length of 3.61 words per expression, and each image contains multiple objects of the same class. It is divided into training, validation, test A and test B, with 120,624, 10,834, 5,657 and 5,095 samples, respectively; RefCOCO+ contains 19,992 images, including 49,856 objects and 141,564 corresponding text expressions. Similar to RefCOCO, these data are collected from MSCOCO and divided into training, validation, test A and test B, using 120,191, 10,758, 5,726 and 4,889 samples, respectively. Unlike RefCOCO, the expressions in RefCOCO+ contain more appearance than absolute position. Unlike RefCOCO, RefCOCO+ prohibits players from using position words to describe targets from the perspective of the computer; the G-Ref dataset is also collected from MSCOCO, containing 26,711 images, 54,822 objects and 104,560 text expressions. Unlike RefCOCO and RefCOCO+ collected based on interactive games, the G-Ref dataset is collected in the form of tasks posted on Amazon Mechanical Turk, and each image contains 2 to 4 objects of the same object class. Due to the lack of time limit, G-Ref has a more complex and complete text expression.

[0070] To better evaluate the performance of the model, we use mIoU and Prec@X as two indicators to evaluate our model. mIoU represents the intersection over union of the predicted results and the mask image, and Prec@X makes a more accurate comparison of the model by measuring the proportion of the test image that is above the threshold X. We set X∈{0.5, 0.6, 0.7, 0.8, 0.9}. This indicator focuses on the positioning ability of the model. We set the image size to 416x416, the text length of RefCOCO, RefCOCO+ and G-Ref to 15, 15 and 20 respectively, use Adam as the optimizer for training, set the initial learning rate to 0.001, set the weight decay coefficient to 0.1 after 30 rounds, and train for a total of 50 rounds to obtain the final referential segmentation model.

[0071] We conducted an ablation experiment on the fusion module, secondary selection mechanism and contrastive attention loss, and the results are shown in Figure 5 It can be seen that the fusion module and the secondary selection mechanism improve the accuracy of the baseline by about 5%, and the secondary selection mechanism improves more, and the proposed contrastive attention loss also optimizes the training accuracy of the model.

[0072] In Figure 6 , rows 1-3 correspond to the test images of RefCOCO, RefCOCO+ and G-Ref respectively. From the text description and segmentation results in figures (a) and (b), we can see that our model focuses on the color, texture and other features of the object. From figures (c) and (d), we can see that the model focuses on the comparative words in the text and can judge the positional relationship of each object. And from figures (e) and (f), we can see that for more complex text expressions, our model can also segment better results, indicating that it is robust to text.

[0073] Figure 7 and Figure 8 From the two indicators of IoU and Prec@X, it shows that the accuracy of our model is superior to previous methods and achieves the best results.

[0074] In summary, in the actual application of referential segmentation, the problems of difficult alignment and interaction between multi-modal features and difficult suppression of background features have not been effectively solved. Therefore, the present embodiment designs a text-guided fusion module and secondary selection mechanism, and obtains a referential segmentation method based on multi-scale feature selective fusion, which provides a high-precision solution for the application fields of referential segmentation such as robot interaction control and intelligent matting.

[0075] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0076] It is to be understood that the application is not limited to the precise construction herein described and as shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is to be indicated by the appended claims, rather than the description and examples.

Claims

1. A method for referential segmentation based on selective fusion of multi-scale features, characterized in that: The segmentation model includes a DarkNet53 backbone network, a bidirectional GRU, a language-driven feature-selective jump module, a secondary selection module, and an image post-processing segmentation module; The method comprises: Step 10: Visual feature extraction. For a given RGB image, first perform cropping and scaling, and then use the DarkNet53 encoder to obtain three different scale visual features F. v1 、F v2 、F v3 ; Step 20: Text feature extraction. For a given text, first segment the text and convert it into a vector form. Then use the bidirectional GRU to extract preliminary text features. Finally, based on the text attention mechanism, perform weighted summation according to the importance of each word to obtain the text feature F. l ; Step 30: First, F is fused by the language-based feature selective jump module. v1 、F v2 、F v3 Select and obtain visual features that are only relevant to the text description, and then jump-connect the selected features through the jump part of the language-based feature selective jump module; The step 30 includes steps 31 to 34: Step 31: F v1 、F v2 、F v3 Input to the fusion part of the language-based feature selective jump module, and F l Perform dot product operation to obtain the features after dot product fusion Among them, W l Indicates that F l Stacked to the same width and height as the visual feature, Represents a 1×1 convolutional layer operation, ⊙ represents a matrix dot multiplication operation, and g represents a Leaky-Relu activation function; Step 32: Input to the spatial fusion branch to obtain the spatial attention map Will Input to the channel fusion branch to obtain the channel attention map W v 、W q 、W z represents a 1×1 convolutional layer operation, F SG Represents the Sigmoid operation, W q represents the Softmax operation, F GP Represents a global pooling operation; Step 33: Multiply the attention map with the visual features to obtain the visual features selected by the channel attention and visual features selected by spatial attention Then we get the visual features that are only related to the text description Step 34: The jump part of the language-based feature selective jump module is used to select Perform jump connection and merge the upsampled features with the previous layer of visual features to obtain multi-scale fused visual features. The features after upsampling of one layer of visual features concat means concatenating features on the channel. Express Perform convolution operation, Indicates the Convolution operation, Up represents upsampling operation; Step 40: Use the secondary selection module to filter and locate the target features to obtain visual features that only contain the target Among them, the fusion feature after the dot product operation of the visual features and text features is once again Among them, W l ' indicates that F l Stacked to the same width and height as the visual feature, W v ′ represents a 1×1 convolutional layer operation; Step 50: Obtain segmentation prediction values ​​through the image post-processing segmentation module; Step 60: Use the training data to train the bidirectional GRU, language-based feature selective jump module, secondary selection module, and image post-processing segmentation module in the segmentation model. The segmentation prediction loss used in the training is Represents the predicted value, and the attention contrast loss used in training is y i represents the true segmentation map, P n represents y i The total loss function used in training is L = αL seg +βL acl , α and β are hyperparameters, representing the respective proportions of segmentation prediction loss and attention contrast loss.

2. The method for referential segmentation based on selective fusion of multi-scale features according to claim 1, characterized in that: The step 10 further includes: Step 11: Crop the given RGB image to a size of 416×416 and then input it into the DarkNet53 backbone network; Step 12: Extract the visual features F of the last layer v1 , the visual feature F of the 152nd layer v2 and the visual features F on the 69th layer v3 ; The Darknet53 backbone network includes a CBL module and a ResX module, wherein the CBL module includes a convolutional layer, a data normalization layer and a Leaky-Relu activation layer, and the ResX module includes a Res1 module, a Res2 module, two Res8 modules and a Res4 module.

3. The method for referential segmentation based on selective fusion of multi-scale features according to claim 1, characterized in that: The step 20 further includes: Step 21: Segment the given text to get T∈[w1,w2,…,w n ], n represents the length of the text; Step 22: Vectorize the word segmentation according to the GloVe model to obtain a text vector of length n, and feed the text vector into the bidirectional GRU network to obtain preliminary text features. in, represents the forward GRU operation, Represents the reverse GRU operation; Step 23: Get text features based on the text attention mechanism 4. The method for referential segmentation based on selective fusion of multi-scale features according to claim 1, characterized in that: The step 50 further comprises: Step 51: F twice The first to fifth parallel operations are performed in parallel, wherein: The first parallel operation is a 1×1 convolution operation; The second parallel operation is a 3×3 convolution operation with a dilation rate of 6 and a padding of 6; The third parallel operation is a 3×3 convolution operation with a dilated convolution rate of 12 and a padding of 12; The fourth parallel operation is a 3×3 convolution with a dilated convolution rate of 18 and a padding of 18; The fifth parallel operation is the pooling operation, which first performs a global average pooling operation, then a 1×1 convolution operation, and finally upsampling to restore to the input size; Step 52: The results of the first to fifth parallel operations are concatenated on the channel, and finally a 1×1 convolution operation is performed to reduce the dimension. Step 53: Iteratively perform upsampling twice on the dimensionality reduction result and then perform convolution processing until the size is comparable to the actual segmentation map size, and obtain the segmentation prediction result.

5. A computer device, characterized in that: include: a memory for storing instructions; as well as A processor, configured to call the instructions stored in the memory to execute the reference segmentation method based on selective fusion of multi-scale features according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that Instructions are stored, and when the instructions are executed by the processor, the reference segmentation method based on selective fusion of multi-scale features according to claims 1-3 is executed.

Images