A multi-scale cross-media information fusion method

Abstract

The present invention belongs to the field of computer vision in deep learning technology, and provides a multi-scale cross-media information fusion method. First, the present invention constructs a feature extraction network, then uses a simple fusion module to obtain initial multimodal information and context information, then extracts information of the input image features at multiple scales and with different receptive fields through a dynamic convolution module, and finally uses a global attention module to use the global context information as the guidance for low-level features, effectively adapting to feature maps at different scales to obtain the final segmentation result. The present invention realizes end-to-end cross-media information deep fusion, has a low model complexity, and can fully and effectively utilize image information and natural language information for image segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, relates to deep learning and computer vision content, and particularly relates to a cross-media information fusion method. Background Art

[0002] In recent years, based on deep learning technology, breakthroughs have been made in single-modal research fields such as language and vision. Long et al. proposed a fully convolutional network for solving semantic segmentation problems in 2015, which directly predicts the labels of each pixel of the input image in an end-to-end manner, providing a new idea for solving semantic segmentation problems. The attention mechanism proposed by Vaswani et al. has achieved very good results in natural language processing problems. As the basic problems of single-modal are gradually solved, researchers have also realized that higher-level artificial intelligence tasks often involve more complex information processing problems across multiple modalities, and cross-modal information processing needs to be studied. Cross-modal research has received more and more attention and has gradually become an important research topic in the next stage of the field of artificial intelligence.

[0003] Early cross-modal algorithms simply fused language and visual features using concatenation and convolution. Hu et al. concatenated the two features and then fused them through convolution. Li et al. gradually combined multi-modal features in the decoder through convolutional LSTM. They did not explicitly consider the relationship between each pixel and each word, nor could they reflect the process of human problem-solving.

[0004] Subsequently, a series of methods using attention mechanisms to model cross-modal relationships have better promoted the fusion between language and visual features. The STEP algorithm proposed by Chen et al. calculates the correlation between each word and each visual region and recursively guides the segmentation of the previous layer using the results of each layer. Huang et al. use a cross-modal fully connected graph model for relationship reasoning, strengthening the most relevant target regions while suppressing irrelevant targets. Shi et al. proposed a keyword-aware network that calculates the similarity between text features and visual features using an attention model and extracts the keywords described in the text. Ye et al. draw on non-local modules to capture the long-term dependencies of multi-modal features. Hui et al. use dependency analysis trees to suppress invalid nodes in the fully connected word graph and at the same time guide the effective multi-modal context of all words and sentences. However, these methods update language and visual features sequentially in a serial manner, and only single-modal features are updated at each specific time, resulting in a delay in the update of features between different modalities and weakening the consistency of multi-modal information expression. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to make up for the deficiencies of the above existing methods, achieve the deep fusion of visual and language features, and well restore image information, and propose a multi-scale cross-media information fusion method, which is applied to referential image segmentation to achieve the purpose of high-precision and high-efficiency segmentation.

[0006] The technical solution of the present invention:

[0007] A multi-scale cross-media information fusion method, the steps are as follows:

[0008] (1) Construct a basic feature extraction network

[0009] The visual feature extraction network uses the pre-trained ResNet101 network structure as the backbone, and extracts the features of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size from different layers as outputs, denoted as And the number of channels corresponding to the output features are 32, 64, 128, 256, and 512 channels respectively. The language feature extraction network first uses WordEmbedding to encode the features corresponding to each word, and the encoded word features are T is the number of words in the sentence, and then uses Bi-GRU to extract the global context features of the language, denoted as h T ;

[0010] (2) Construct a feature initial fusion module

[0011] Insert the language context feature h T After the image encoder features V3, V4, and V5 respectively, the ResNe101 is transformed into a multi-modal feature encoder, which can make full use of the powerful data fitting ability of the deep convolutional neural network model and realize the deep interweaving of cross-modal features. Here, a simple concatenation operation is first used to obtain the initial multi-modal features:

[0012]

[0013] Among them, Concat represents the concatenation operation, Represents the feature vector of the encoder feature V i At position p. Is an 8-dimensional spatial coordinate, which can supplement the spatial information of the network.

[0014] Next, calculate the language context of each position according to the feature m p :

[0015] α p,t =m p ·e t ,

[0016]

[0017]

[0018] Among them, represents the correlation between each pixel and each word, and the position - adaptive language context l can be obtained by weighted fusion of its language features. p ;

[0019] (3) Construct a dynamic convolution module

[0020] The updated context information L at all positions obtained after the initial fusion is fed into the dynamic convolution module to generate convolution kernels with three dilation rates, which are 1×1, 3×3, and 5×5 respectively. These three convolution kernels with different dilation rates are used as the convolution kernels of the dynamic filter to convolve the multi - modal features M in the down - sampling process, and then the obtained features are concatenated along the channel direction to generate the final multi - modal features. This realizes the extraction of information of the input image features under multi - scales and different receptive fields, obtains richer and more diverse context information, and provides a more effective feature representation for subsequent feature processing. Stacking multiple convolution kernels is also very efficient in terms of calculation. Although it brings a small amount of additional calculation, it significantly improves the expressive ability.

[0021] (4) Construct a global attention module

[0022] The features obtained from the encoder are fed into the global attention module. A 3×3 convolution operation is performed on the low - level features to reduce the number of channels of the feature map. The high - level features are successively passed through 1×1 convolution, batch normalization, and linear transformation operations, and then multiplied by the low - level features to obtain the weighted low - level features. This feature is added to the high - level features and a progressive up - sampling process is carried out. Using the global context information as the guidance for the low - level features through the global pooling process can not only effectively adapt to the feature maps at different scales, but also provide guidance information for the low - level feature maps in a simple way. Specifically, the rich semantic information contained in the high - level features is used to guide the low - level features to better restore pixel information.

[0023] (5) Training method

[0024] Use ResNet101 as the backbone network. First, pre - train it on the image classification dataset ImageNet and save the pre - trained network weight parameters; then train the whole network on UNC, UNC +, G - Ref, and ReferIt respectively; use the cross - entropy loss function as the objective function for optimization, use the SGD optimization algorithm, set the momentum to 0.9, the weight decay rate to 0.0005, the batch size to 8, the initial learning rate to 0.001, and reduce the learning rate to 0.9 of the original after 50,000 iterations, and perform a total of 100,000 iterations.

[0025] Advantages of the present invention: The present invention makes full use of the complementary information contained in images and language, converts the visual encoder based on a convolutional neural network into a multi-modal feature encoder, establishes the connection between multi-modal features of different scales, and realizes the deep interactive learning between the two modal features through dynamic convolution. In addition, due to the inevitable loss of a lot of fine boundary information during multiple downsamplings, the global attention module uses the rich category information contained in the high-level features to guide the low-level features to better restore pixel information. When applied to referential image segmentation, good segmentation effects are achieved on each dataset. Description of the Drawings

[0026] Figure 1 is the overall architecture diagram of the multi-scale cross-media information fusion method. The top row represents the language feature extraction branch Bi-GRU, the bottom row represents the backbone network for image feature extraction, and the boxes of different sizes in the bottom row represent the feature maps of different scales of the decoder. The decoder feature map is finally upsampled by GAU to obtain the final segmentation result;

[0027] Figure 2 is the dynamic convolution module, where × represents pixel multiplication operation;

[0028] Figure 3 is the schematic diagram of the global attention module, where × represents pixel multiplication operation and + represents pixel addition operation;

[0029] Figure 4 are examples of the input image and query statement;

[0030] Figure 5 are examples of the referential segmentation results. Detailed Embodiments

[0031] The following further describes the detailed embodiments of the present invention in conjunction with the drawings and technical solutions.

[0032] A multi-scale cross-media information fusion method, the steps are as follows:

[0033] (1) Construct a basic feature extraction network

[0034] Taking Figure 4 as the input image, as shown in Figure 1 , the ResNet101 network is used as the visual feature extraction network. ResNet101 contains five convolutional blocks, and the features extracted by these five convolutional blocks are respectively represented as The corresponding referential expression sentence is "the child wearing red clothes". First, WordEmbedding is used to encode the features corresponding to each word in the referential expression sentence, and the encoded word features are Let \(T\) be the number of words in the sentence, and then use a Bi - GRU to extract the global context features of the language, denoted as \(h\). T 。

[0035] (2) Construct the initial feature fusion module

[0036] As Figure 1 shown in the second row, insert the language feature \(h\) T after the image features \(V_3\), \(V_4\), and \(V_5\) respectively, and obtain the initial multi - modal features through a simple concatenation operation:

[0037]

[0038] where Concat represents the concatenation operation, represents the feature vector of the encoder feature \(V\) i at position \(p\). is an 8 - dimensional spatial coordinate, which can supplement the spatial information of the network.

[0039] Next, calculate the language context at each position according to the feature \(m\) p :

[0040] \(\alpha\) p,t = m p · e t ,

[0041]

[0042]

[0043] where, represents the correlation between each pixel and each word, and the position - adaptive language context \(l\) p can be obtained by weighted fusion of the language features with it.

[0044] (3) Construct the dynamic convolution module

[0045] As Figure 2 shown, \(L\) represents the updated context information at all positions obtained through the initial fusion module. Generate three different dilation rate convolutional kernels from \(L\), namely \(1\times1\), \(3\times3\), and \(5\times5\). Use these three different dilation rate convolutional kernels as the convolutional kernels of the dynamic filter to convolve the multi - modal feature \(M\) in the down - sampling process, and then concatenate the obtained features along the channel direction to generate the final multi - modal feature. Corresponding to Figure 1 the DCM module, it realizes the extraction of information of the input image features under multi - scale and different receptive fields, obtains more rich and diverse context information, and provides a more effective feature representation for subsequent feature processing.

[0046] (4) Construct the global attention module

[0047] As Figure 3 shown, the features obtained from the encoder are fed into the global attention module GAU. Taking D4 and D5 shown in Figure 1 as an example, D4 corresponds to low-level features and D5 corresponds to high-level features. A 3×3 convolution operation is performed on the low-level feature D4 to reduce the number of channels of the feature map. The high-level feature D5 goes through 1×1 convolution, batch normalization, and linear transformation operations in sequence, and then is multiplied by the low-level feature to obtain the weighted low-level feature. The weighted low-level feature is added to the high-level feature and a progressive upsampling process is carried out. For the global pooling module, using the global context information as the guidance for the low-level features can not only effectively adapt to feature maps at different scales, but also provide guidance information for the low-level feature maps in a simple way. Figure 5 It is the corresponding reference segmentation result.

[0048] (5) Training method

[0049] Use ResNet101 as the backbone network. First, pre-train it on the image classification dataset ImageNet and save the pre-trained network weight parameters; then train the entire network on UNC, UNC+, G-Ref, and ReferIt respectively; use the cross-entropy loss function as the objective function for optimization, use the SGD optimization algorithm, set the momentum to 0.9, the weight decay rate to 0.0005, the batch size to 8, the initial learning rate to 0.001, and reduce the learning rate to 0.9 of the original after 50,000 iterations, and perform a total of 100,000 iterations.

Claims

1. A multi-scale cross-media information fusion method, characterized in that The network framework adopted by the multi-scale cross-media information fusion method includes a visual and language feature extraction network, a multi-modal feature fusion module, and a global attention module; The multi-modal feature fusion module includes a feature initial fusion module and a dynamic convolution module; among them, the visual and language feature extraction networks are respectively used to extract the features of two modalities of the input image and the reference expression sentence. The two extracted features are sent into the multi-modal feature fusion module. First, a simple multi-modal feature expression is obtained through the feature initial fusion module, and then more rich and diverse context information is further obtained through the dynamic convolution module. Finally, more accurate segmentation information is obtained through the global attention module; The specific steps are as follows: (1) Construct a basic feature extraction network The visual feature extraction network uses the ResNet101 network structure as the backbone, and extracts the image encoder features of 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size from different layers as outputs, denoted as and the number of channels corresponding to the output features is 32, 64, 128, 256, and 512 channels respectively; The language feature extraction network first uses WordEmbedding to encode the features corresponding to each word. The encoded word features are where T is the number of words in the sentence, and then uses Bi-GRU to extract the global context features of the language, denoted as h T ; (2) Construct a feature initial fusion module Insert the global context feature h of the language T after the image encoder features V3, V4, and V5 respectively, so that ResNe101 is transformed into a multi-modal feature encoder; First, a concatenation operation is used to obtain an initial multi-modal feature encoder: Among them, Concat represents a concatenation operation, represents the feature V of the image encoder i feature vector at position p; is an 8-dimensional spatial coordinate used to supplement the spatial information of the network; Next, according to feature m p calculate the language context at each of its positions: α p,t = m p ·e t , Among them, represents the correlation between each pixel and each word, and obtains the position-adaptive language context l by weighted fusion of its language features p ; (3) Construct a dynamic convolution module Let \(M\) and \(L\) denote the sets of feature vectors \(m\) and \(l\) at all positions respectively. p And \(l\) p The updated context information \(L\) at all positions obtained through the feature initial fusion module first generates convolution kernels with three different dilation rates through the dynamic convolution module. The dilation rates are \(1\times1\), \(3\times3\), and \(5\times5\). Then, these three convolution kernels are used as the convolution kernels of the dynamic filter to convolve the multi-modal feature \(M\) in the downsampling process. Finally, the obtained features are concatenated along the channel direction to generate the final multi-modal feature. Among them, G j (j ∈ {1, 3, 5}) represents a series of convolution operations for generating convolution kernels with dilation rate j, represents an adaptive convolution operation; (4) Construct a global attention module The multi-modal features obtained by dynamic convolution are fed into the global attention module. First, for the low-level features a 3×3 convolution operation is performed to reduce the number of channels of the feature map; then, for the high-level features they are successively passed through 1×1 convolution, batch normalization, and linear transformation operations, and then multiplied by the low-level features to obtain the weighted low-level features. Finally, the weighted low-level features are added to the high-level features and a progressive upsampling process is carried out to obtain the final segmentation features Among them, g(·) represents a 3×3 convolution operation, f(·) represents global average pooling, 1×1 convolution, batch normalization, and linear transformation operations, and upsample represents upsampling; specifically, through the global attention module, the global context information is used as the guidance for the low-level features, and the rich semantic information contained in the high-level features is used to guide the low-level features to better restore the pixel information; (5) Training method Use ResNet101 as the backbone network. First, pre-train it on the image classification dataset ImageNet and save the pre-trained network weight parameters; then train the entire network on UNC, UNC+, G-Ref, and ReferIt respectively; optimize with the cross-entropy loss function as the objective function, use the SGD optimization algorithm, set the momentum to 0.9, the weight decay rate to 0.0005, the batch size to 8, the initial learning rate to 0.001, and reduce the learning rate to 0.9 of the original after 50,000 iterations, and perform a total of 100,000 iterations.

Images