Visual feature and text feature fused position coding method, system and device

Abstract

The invention provides a visual feature and text feature fused position coding method, system and device. The method comprises the steps of obtaining an image and a text sequence in to-be-processed data; a convolutional neural network is adopted for the image to obtain multi-scale features of the image, multi-scale feature fusion operation is performed on the multi-scale features, and a compressed feature map is obtained; image domain position coding is carried out on the compressed feature map by adopting image domain position coding to obtain image feature embedding; performing text embedding mapping on the text sequence to obtain text feature embedding; image feature coding and text feature coding are uniformly achieved for image feature embedding and text feature embedding through position coding, a fused position coding vector is generated, and the accuracy of the image information position can be improved through the technical scheme.

Description

Technical Field

[0001] This document relates to the technical field of image-text feature fusion, and in particular to a position encoding method, system and device for fusing visual features with text features. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning in the field of multimedia processing, Multi-modal Image-Text Large Models have become an important bridge between vision and language, greatly promoting the development of cross-modal understanding and generation technology. Such models can parse the visual information in the image and generate related natural language descriptions, or generate corresponding images based on the text content, showing great application potential.

[0003] However, the current large-scale model of image-text modality mainly relies on the visual Transformer architecture in processing visual information. Although this architecture has shown advantages in capturing global dependencies and modeling complex visual patterns, its computational complexity is high and its processing efficiency is relatively low, especially when processing high-resolution images. In addition, the Transformer architecture is still insufficient in processing local details and texture features compared to traditional convolutional neural networks, which limits the performance of the model on fine visual tasks.

[0004] In view of this, how to effectively fuse convolutional features to make up for the shortcomings of visual transformers in efficiency and local feature extraction has become a key bottleneck for the further development of large-scale image-text modal models. Convolutional neural networks can efficiently extract multi-scale local features from images through their hierarchical convolution kernel design. These features are essential for understanding and describing the details in the image. Therefore, exploring a method to effectively integrate convolutional features into the generation process of large-scale image-text modal models can not only improve the visual understanding ability of the model, but also significantly enhance the generation efficiency and effect of the model, which has important research significance and application value. Summary of the invention

[0005] The present invention provides a method, system and device for position coding by fusing visual features with text features, aiming to solve the above-mentioned problem.

[0006] The embodiment of the present invention provides a position encoding method for fusing visual features with text features, including:

[0007] S1, obtaining the image and text sequences in the data to be processed;

[0008] S2. Using a convolutional neural network to obtain multi-scale features of the image, and performing a multi-scale feature fusion operation on the multi-scale features to obtain a compressed feature map;

[0009] S3, performing image domain position coding on the compressed feature map by using image domain position coding to obtain image feature embedding;

[0010] S4, performing text embedding mapping on the text sequence to obtain text feature embedding;

[0011] S5. Position coding is used to uniformly implement image feature coding and text feature coding for the image feature embedding and the text feature embedding, so as to generate a fused position coding vector.

[0012] The embodiment of the present invention provides a position coding system for fusing visual features with text features, including:

[0013] A data acquisition module, used to acquire image and text sequences in the data to be processed;

[0014] A multi-scale feature fusion module, used to obtain multi-scale features of the image using a convolutional neural network, perform a multi-scale feature fusion operation on the multi-scale features, and obtain a compressed feature map;

[0015] An image position coding module, which uses image domain position coding to perform image domain position coding on the compressed feature map to obtain image feature embedding;

[0016] A text mapping module, used for performing text embedding mapping on the text sequence to obtain text feature embedding;

[0017] The image-text fusion module is used to use position coding to uniformly implement image feature coding and text feature coding for the image feature embedding and the text feature embedding, and generate a fused position coding vector.

[0018] One or more embodiments of the present specification provide an electronic device including:

[0019] processor; and,

[0020] A memory is arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the position encoding method for fusing scaled visual features with text features as described above.

[0021] One or more embodiments of the present specification provide a storage medium for storing computer executable instructions, which, when executed, implement the steps of the position encoding method for fusing scaled visual features with text features as described above.

[0022] By adopting the embodiment of the present invention, the efficiency of image feature extraction is improved by utilizing a convolutional network. At the same time, the image features are encoded by image domain position to ensure that the features are contained in the image, thereby improving the global information perception of the subsequent generative large model. After unified encoding after the fusion of graphic and text features, the position of the image features in the text sequence is determined, thereby forming a two-level encoding form and improving the accuracy of the image information position. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 It is a flow chart of a position encoding method for fusing scaled visual features with text features according to an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of a specific implementation of the position coding method for fusing scaled visual features with text features according to an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of multi-scale convolution feature fusion of the present embodiment;

[0027] Figure 4 A schematic diagram of image domain position coding according to an embodiment of the present invention;

[0028] Figure 5 Schematic diagram of a position encoding system for fusing scaled visual features with text features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0030] Method Embodiment

[0031] According to an embodiment of the present invention, a position encoding method for fusing scaled visual features with text features is provided. Figure 1Flow chart of the position encoding method for fusing scaled visual features with text features according to an embodiment of the present invention. Figure 1 As shown, the position encoding method for fusing scaled visual features with text features in an embodiment of the present invention specifically includes:

[0032] S1, obtaining the image and text sequences in the data to be processed;

[0033] S2. Using a convolutional neural network to obtain multi-scale features of the image, and performing a multi-scale feature fusion operation on the multi-scale features to obtain a compressed feature map;

[0034] Figure 3 Schematic diagram of a specific implementation of the position coding method for fusing scaled visual features with text features according to an embodiment of the present invention. Figure 2 It can be seen that the multi-scale feature fusion operation on the multi-scale features in the embodiment of the present invention specifically includes:

[0035] There are three parts: horizontal semi-global feature fusion, vertical semi-global feature fusion, and bidirectional global feature fusion. The horizontal semi-global feature fusion feature is compressed to [1, height, channel]; the vertical semi-global feature fusion feature is compressed to [widht, 1, channel]; the bidirectional global feature fusion compresses the horizontal and vertical feature fusions to a vector with a dimension of [1, 1, channel]. The compression method can use unidirectional average pooling and unidirectional convolution.

[0036] The convolution feature is the result of the intermediate layer of the convolutional network. The specific number of layers can be selected, and the dimension is [widht, height, channel]. After feature fusion, the feature dimension is [widht+1, height+1, channel].

[0037] S3, performing image domain position coding on the compressed feature map by using image domain position coding to obtain image feature embedding;

[0038] Figure 4 This is a schematic diagram of image domain position coding according to an embodiment of the present invention. Figure 4 It can be seen that image domain position coding includes two dimensions: horizontal coding and vertical coding, where:

[0039] The calculation formula for horizontal coding is: sin(x / (2^i));

[0040] The vertical coding calculation formula is: cos(x / (2^j);

[0041] Among them, x is a trainable parameter, i is the horizontal position of the image where the feature is located, and j is the vertical position.

[0042] The encoding dimension is [widht+1, height+1, 2]. After concatenation, the generated feature is [widht+1, height+1, channel+2]. The feature dimension of channel+2 needs to be consistent with the text feature (the common dimension is 768, so the channel dimension can be 766, which can be changed). After stretching, a feature vector with the dimension of [(widht+1)*(height+1), channel+2] can be obtained.

[0043] S4, performing text embedding mapping on the text sequence to obtain text feature embedding;

[0044] For the input text, a vector with a dimension of [seq_length, emb_dim] can be obtained through the commonly used word embedding model, where emb_dim is commonly 786 and seq_length is related to the length of the text input.

[0045] S5. Position coding is used to uniformly implement image feature coding and text feature coding for the image feature embedding and the text feature embedding, so as to generate a fused position coding vector.

[0046] Concatenate the text vector of dimension [seq_length,emb_dim] with the image feature vector of dimension [(widht+1)*(height+1),channel+2]. As mentioned above, channel+2 must be the same as emb_dim. We can obtain a vector of dimension [seq_length+(widht+1)*(height+1),emb_dim].

[0047] Use position encoding to uniformly encode image features and text features, and generate a fused position encoding vector. After that, it can be processed according to the generative text model.

[0048] Figure 2 This is a schematic diagram of a specific implementation of a position coding method for fusing scaled visual features with text features in an embodiment of the present invention, wherein image domain position coding is the core of this patent. This method generates a feature vector with image domain coding based on convolutional features, and can use convolutional features for conventional generative large model position coding methods, such as rotational position coding.

[0049] The image multi-scale convolution feature fusion is oriented towards the characteristics of multi-layer position coding features. It uses local image information to generate semi-global and global information, and combines it with image domain position coding to enhance the subsequent generative large model's perception of global information.

[0050] The image-text feature fusion encoding mainly describes how the generated image features are fused with the conventional text embedding features.

[0051] The embodiments of the present invention have the following beneficial effects:

[0052] By adopting the embodiment of the present invention, the efficiency of image feature extraction is improved by utilizing a convolutional network. At the same time, the image features are encoded by image domain position to ensure that the features are contained in the image, thereby improving the global information perception of the subsequent generative large model. After unified encoding after the fusion of graphic and text features, the position of the image features in the text sequence is determined, thereby forming a two-level encoding form and improving the accuracy of the image information position.

[0053] System Example

[0054] According to an embodiment of the present invention, a position encoding system for fusing scaled visual features with text features is provided. Figure 5 Schematic diagram of a position encoding system for fusing scaled visual features with text features according to an embodiment of the present invention. Figure 5 As shown, in Embodiment 5 of the present invention, the position coding system for fusing scaled visual features with text features in the embodiment of the present invention specifically includes:

[0055] A data acquisition module 50 is used to acquire image and text sequences in the data to be processed;

[0056] An image multi-scale convolution feature fusion module 52, configured to obtain multi-scale features of the image using a convolutional neural network, and perform a multi-scale feature fusion operation on the multi-scale features to obtain a compressed feature map;

[0057] An image domain position coding module 54, which uses image domain position coding to perform image domain position coding on the compressed feature map to obtain image feature embedding;

[0058] A text mapping module 56, configured to perform text embedding mapping on the text sequence to obtain text feature embedding;

[0059] The image-text feature fusion coding module 58 is used to use position coding to uniformly implement image feature coding and text feature coding for the image feature embedding and the text feature embedding, and generate a fused position coding vector.

[0060] The image multi-scale convolution feature fusion module 52 performs a multi-scale feature fusion operation on the multi-scale features, specifically including:

[0061] The multi-scale features of the image are subjected to horizontal semi-global feature fusion, vertical semi-global feature fusion, and bidirectional global feature fusion respectively.

[0062] The horizontal semi-global feature fusion compresses the image into a vector with a dimension of [1, height, channel], the vertical semi-global feature fusion compresses the image into a vector with a dimension of [widht, 1, channel], and the bidirectional global feature fusion compresses the horizontal semi-global feature fusion and the vertical semi-global feature fusion into a vector with a dimension of [1, 1, channel], wherein height represents the height of the image, width represents the width of the image, and channel represents the number of channels of the image feature.

[0063] The image domain position encoding module 54 is specifically used for:

[0064] The compressed feature map is horizontally encoded and vertically encoded, wherein:

[0065] The horizontal coding is obtained by the following formula:

[0066] sin(x / (2^i)) (1);

[0067] The vertical coding is obtained by the following formula:

[0068] cos(x / (2^j)(2);

[0069] Among them, x is a trainable parameter, i is the horizontal position of the image where the feature is located, and j is the vertical position.

[0070] Device Example 1

[0071] According to an embodiment of the present invention, an electronic device is provided, including:

[0072] processor; and,

[0073] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps as described in the above method embodiment.

[0074] Device Example 2

[0075] According to an embodiment of the present invention, a storage medium is provided for storing computer executable instructions, wherein the computer executable instructions, when executed, implement the steps described in the above method embodiment.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A position encoding method for fusing visual features with text features, characterized in that include: S1, obtaining the image and text sequences in the data to be processed; S2. Using a convolutional neural network to obtain multi-scale features of the image, and performing a multi-scale feature fusion operation on the multi-scale features to obtain a compressed feature map; S3, performing image domain position coding on the compressed feature map by using image domain position coding to obtain image feature embedding; S4, performing text embedding mapping on the text sequence to obtain text feature embedding; S5. Position coding is used to uniformly implement image feature coding and text feature coding for the image feature embedding and the text feature embedding, so as to generate a fused position coding vector.

2. The method according to claim 1, characterized in that The performing a multi-scale feature fusion operation on the multi-scale features to obtain a compressed feature map specifically includes: The multi-scale features of the image are subjected to horizontal semi-global feature fusion, vertical semi-global feature fusion, and bidirectional global feature fusion respectively.

3. The method according to claim 2, characterized in that The horizontal semi-global feature fusion compresses the image into a vector with a dimension of [1, height, channel], the vertical semi-global feature fusion compresses the image into a vector with a dimension of [widht, 1, channel], and the bidirectional global feature fusion compresses the horizontal semi-global feature fusion and the vertical semi-global feature fusion into a vector with a dimension of [1, 1, channel], wherein height represents the height of the image, width represents the width of the image, and channel represents the number of channels of the image feature.

4. The method according to claim 1, characterized in that: The image domain position coding specifically includes two dimensions: horizontal coding and vertical coding, wherein: The horizontal coding is obtained by the following formula: sin(x / (2^i))(1); The vertical coding is obtained by the following formula: cos(x / (2^j)(2); Among them, x is a trainable parameter, i is the horizontal position of the image where the feature is located, and j is the vertical position.

5. A position encoding system for fusing visual features with text features, characterized in that: include: A data acquisition module, used to acquire image and text sequences in the data to be processed; An image multi-scale convolution feature fusion module, used to obtain multi-scale features of the image using a convolutional neural network, perform a multi-scale feature fusion operation on the multi-scale features, and obtain a compressed feature map; An image domain position coding module, which uses image domain position coding to perform image domain position coding on the compressed feature map to obtain image feature embedding; A text mapping module, used for performing text embedding mapping on the text sequence to obtain text feature embedding; The image-text feature fusion coding module is used to use position coding to uniformly implement image feature coding and text feature coding for the image feature embedding and the text feature embedding, and generate a fused position coding vector.

6. The system according to claim 5, characterized in that The image multi-scale convolution feature fusion module performs a multi-scale feature fusion operation on the multi-scale features, specifically comprising: The multi-scale features of the image are subjected to horizontal semi-global feature fusion, vertical semi-global feature fusion, and bidirectional global feature fusion respectively.

7. The system according to claim 6, characterized in that The horizontal semi-global feature fusion compresses the image into a vector with a dimension of [1, height, channel], the vertical semi-global feature fusion compresses the image into a vector with a dimension of [widht, 1, channel], and the bidirectional global feature fusion compresses the horizontal semi-global feature fusion and the vertical semi-global feature fusion into a vector with a dimension of [1, 1, channel], wherein height represents the height of the image, width represents the width of the image, and channel represents the number of channels of the image feature.

8. The system according to claim 5, characterized in that The image domain position encoding module is specifically used for: The compressed feature map is horizontally encoded and vertically encoded, wherein: The horizontal coding is obtained by the following formula: sin(x / (2^i))(1); The vertical coding is obtained by the following formula: cos(x / (2^j)(2); Among them, x is a trainable parameter, i is the horizontal position of the image where the feature is located, and j is the vertical position.

9. An electronic device, comprising: processor; as well as, A memory arranged to store computer executable instructions, wherein when the computer executable instructions are executed, the processor performs the steps of the position encoding method for fusing scaled visual features with text features as claimed in any one of claims 1 to 4.

10. A storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed, implement the steps of the position encoding method for fusing scaled visual features with text features as described in any one of claims 1 to 4.

Images