Vision-language combined ocean ship positioning method based on Transform

Abstract

The invention discloses a vision-language combined ocean ship positioning method based on Transform, and belongs to the field of computer vision and natural language processing. Comprising three steps of data filtering, attribute extraction and expression generation to construct a new ocean remote sensing ship image vision-language combined positioning data set, and at the same time, a visual feature enhancement module is utilized to enhance visual features through language guidance before a vision-language fusion module; on the basis, language features are expanded and then subjected to Concat with visual features, spatial information of the visual features is reserved, feature fusion is performed by using a multi-head self-attention mechanism of Transform, interaction and integration of cross-modal information are realized, and a new loss calculation mode is introduced to enable loss calculation to be more accurate; and finally, the efficiency and accuracy of vision-language combined positioning of the marine ship are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and natural language processing, and specifically relates to a Transformer-based vision-language combined ocean ship positioning method. Background Art

[0002] Visual-language joint localization refers to locating an object in an image under the guidance of natural language given an image and a text description of an object in it. With the continuous advancement of marine remote sensing technology, the resolution of marine images is increasing day by day. Identifying specific ships in marine remote sensing images plays an increasingly important role in marine monitoring and management. It has broad application prospects in the fields of military target positioning, port management, marine environmental protection, shipping management, maritime rescue, marine monitoring and safety.

[0003] Although previous studies have involved multimodal localization methods for visual language, joint localization methods for marine remote sensing images and text are still scarce, and related datasets are very limited. In addition, previous visual-language joint localization methods usually only perform simple interactions between visual features and text features in the fusion module, failing to fully utilize text features. At the same time, these methods often fuse visual features by reducing dimensionality, resulting in the loss of spatial information.

[0004] At present, most solutions to the above problems are centered around building targeted lightweight deep learning algorithms, which have not been extended to complex ocean remote sensing ship images and have limitations. In addition, existing methods attempt to solve the above problems through multi-level feature extraction and more complex interaction mechanisms, but due to the compatibility and consistency issues between different types of features, the overall performance and stability of the model will still be affected. Summary of the invention

[0005] In view of the shortcomings of the background technology, the present invention proposes a Transformer-based vision-language joint ocean ship positioning method, which utilizes the Transformer's self-attention mechanism to collect information within and between modalities, and improves it on this basis, and uses the Transformer's self-attention mechanism to fuse the features of each modality, so that it can accurately extract the features of the ship referred to by the text, and perform regression to predict the ship's coordinates, thereby improving the accuracy of ocean ship positioning.

[0006] In a first aspect, the present invention provides a Transformer-based vision-language combined ocean ship positioning method, comprising: Step S1, collecting ocean remote sensing image data and preprocessing it to obtain an ocean ship positioning data set; Step S2, constructing a Transformer-based vision-language joint ocean ship positioning model, the model comprising a vision module, a language module, a visual feature enhancement module, a vision-language fusion module and a prediction head module; Step S3, inputting the sample data in the ocean ship positioning data set into the model, calculating the total loss function value, performing back propagation, optimizing the connection weights through the selected optimizer and model parameters, and obtaining a trained vision-language joint ocean ship positioning model; Step S4, input the ocean remote sensing ship image and the text description corresponding to the specific ship into the trained visual-language joint ocean ship positioning model, and output the specific coordinates of the ship corresponding to the text description.

[0007] Preferably, step S1 comprises: Step S11, data filtering: filtering the image data related to the ocean remote sensing ship, deleting other types of data, irregular data and erroneous data, and obtaining image data suitable for vision-language combined ocean ship positioning; Step S12, attribute extraction; the attribute extraction determines a set of attributes applicable to the marine remote sensing ship data, extracts the attributes of the marine remote sensing ship according to different attribute extraction methods, and obtains a marine remote sensing ship attribute dictionary; Step S13, expression generation; the expression generation determines a set of text description sentence templates suitable for marine remote sensing ship data, selects appropriate sentence templates to fill in attribute information according to the marine remote sensing ship attribute dictionary, obtains the text expression of the marine remote sensing ship, saves all generated text expressions into a list, writes them into the corresponding label file, and obtains the visual-language joint marine ship positioning data set.

[0008] Preferably, the visual module in step S2 takes the given ocean remote sensing ship image as input and uses the backbone network to extract features to generate a two-dimensional feature map , and then use convolution to reduce The channel dimension of Flattening, using sinusoidal spatial position encoding as The complement of , and use the visual Transformer to perform global visual context reasoning, the output is High-level visual features with the same shape .

[0009] Preferably, the language module in step S2 includes a token embedding layer and a language transformer, and for the text expression, each word is embedded into a one-hot embedding vector, each one-hot vector is converted into a language tag, and a [CLS] tag and a [SEP] tag are appended, and a BERT model with 6 Transformer encoder layers containing pre-trained weights is used to extract word-level text features and sentence-level text features, and the language tags are used as input to the language transformer, and high-level language features are generated. .

[0010] Preferably, the visual feature enhancement module in step S2 includes two parts: a language-guided visual feature enhancement module and a multimodal feature stacking module; The language-guided visual feature enhancement module is used to extract high-level visual features from the visual module. And high-level language features extracted by the language module , apply a linear projection layer to transform the language features Projection to visual features Update visual features among features with the same dimension Dimension, will As Query, The multi-head self-attention layer of the Transformer is used as the key and value for calculation, and then the calculation result is mapped through a linear layer and combined with the initial Add the residuals to get the high-level visual features enhanced by language guidance ; The multimodal feature stacking module combines high-level language features After the steps of dimension transformation, dimension increase, replication, and reshape, a new language feature tensor is obtained after expansion. , the high-level visual features enhanced by language guidance With the new language feature tensor Perform splicing on the channel dimension to obtain the spliced ​​features .

[0011] Preferably, the visual language fusion module in step S2 is used to first randomly initialize a It is used to capture the global information of multimodal fusion, further integrate the visual features and text features through the multi-head self-attention mechanism of Transformer, and then As Query, the original As Key and Value, through the multi-head self-attention mechanism Each feature vector in collects visual language fusion information, and the obtained output and the original Add the residuals to get the new .

[0012] Preferably, the prediction head module in step S2 is used to utilize the visual language fusion module As input to the prediction head, a regression module is attached to On the token, the regression module is implemented by an MLP with two hidden layers with ReLU activations and a linear output layer, outputting the ship coordinates.

[0013] Preferably, in step S3, the total loss function value of the model is calculated Including smooth loss , broad sense loss and loss , the total loss function of the model is The calculation formula is:

[0014] in Weight factor used to balance these losses.

[0015] In a second aspect, the present invention further proposes a computer device comprising a memory, a processor, and program instructions stored in the memory for execution by the processor, wherein the processor executes the program instructions to implement the above-mentioned Transformer-based vision-language joint ocean ship positioning method.

[0016] In a third aspect, the present invention proposes a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the above-mentioned Transformer-based vision-language combined ocean ship positioning method is implemented.

[0017] Compared with the prior art, the present invention proposes a new Transformer-based visual-language joint marine ship positioning method. The method constructs a new visual-language joint marine ship positioning dataset on the original marine remote sensing ship-related image data, and at the same time, makes full use of language information and retains the spatial information of visual features. Specifically, firstly, a new dataset construction method is designed, including three steps of data filtering, attribute extraction and expression generation, to construct a new marine remote sensing ship image visual-language joint positioning dataset, and at the same time, a visual feature enhancement module is used to enhance the visual features through language guidance before the visual-language fusion module. On this basis, the language features are expanded and concat with the visual features to retain the spatial information of the visual features, and then the multi-head self-attention mechanism of the Transformer is used for feature fusion to realize the interaction and integration of cross-modal information, and a new loss calculation method is introduced to make the loss calculation more accurate; finally, the efficiency and accuracy of the visual-language joint positioning of marine ships are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an overall flow chart of a Transformer-based vision-language combined ocean ship positioning method in the present invention.

[0019] Figure 2 This is a model framework diagram of the Transformer-based vision-language joint ocean ship positioning method proposed in the present invention.

[0020] Figure 3 This is the processing flow chart of the visual feature enhancement module.

[0021] Figure 4 This is the processing flow chart of the visual language fusion module. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to further clearly and completely describe the technical solutions in the embodiments of the present application. It should be noted that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0023] In order to make the invention objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application are further described in detail in conjunction with the drawings in the specification: In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the advantages of the present invention will be further illustrated by comparing the embodiments in conjunction with the drawings and specific implementation methods.

[0024] In the first embodiment, the present invention proposes a Transformer-based vision-language combined ocean ship positioning method, the flow chart of which is as follows: Figure 1 As shown, the steps of this method are described in detail below: Step S1, collecting image data related to ocean remote sensing ships and constructing a visual-language joint positioning dataset for ocean remote sensing ship images; Specifically, the construction of the marine remote sensing ship-related image data into a visual-language joint positioning marine ship dataset includes three steps: data filtering, attribute extraction, and expression generation: Step S11, data filtering, removes all categories except ships, deletes data with a very small proportion of ship categories, deletes data with a bounding box proportion of less than 0.03, and removes data with incorrect annotations, such as bounding box coordinates or .

[0025] Step S12, attribute extraction, determines a set of attributes suitable for marine remote sensing ship data, the ship category attribute is obtained from the original data set, the ship size attribute is obtained by the ratio with the original image, and the ship absolute position attribute is obtained by the ship The coordinates of the center point are calculated from the coordinates, the relative position attributes of the ships are calculated based on the orientation of the two ship center points, the relative size attributes of the ships are calculated based on the ratio of the current ship pixel area size to the target ship pixel area size, and the ship color is extracted using perspective transformation and HSV color space conversion. If the original data set also contains other information about the ship's position, we will also extract it into the attributes and put the extracted attributes into the dictionary.

[0026] Step S13, expression generation, determine a set of text description sentence templates suitable for marine remote sensing ship data, select appropriate sentence templates to fill in attribute information according to the marine remote sensing ship attribute dictionary, and for each ship, perform conditional checks on the uniqueness of its attributes such as category, size, boundary block and relative position to determine the minimum number of attributes required to locate the ship, and use the corresponding sentence templates to generate corresponding text expressions, save all generated expressions into a list, write them into the corresponding label file, and obtain a visual-language joint positioning marine ship dataset.

[0027] Step S2, construct a Transformer-based vision-language joint positioning model for marine ships, which includes five modules: vision module, language module, visual feature enhancement module, vision-language fusion module and prediction head module. The entire model framework is as follows Figure 2 As shown; Specifically, the visual module takes the given ocean remote sensing ship image as input and uses the backbone network to extract features to generate a two-dimensional feature map , and then use 1×1 convolution to reduce The channel dimension of Flattening, in order to make the visual Transformer position-sensitive, using sinusoidal spatial position encoding as The visual Transformer is used to perform global visual context reasoning, where the encoder layer of the visual Transformer is 3 layers and the output is High-level visual features with the same shape .

[0028] Furthermore, the language module, including a token embedding layer and a language transformer, embeds each word into a one-hot embedding vector for text expressions, converts each one-hot vector into a language tag, and appends the [CLS] tag and the [SEP] tag. In order to obtain local semantic information and global context information, a BERT model with 6 Transformer encoder layers containing pre-trained weights is used to extract word-level text features and sentence-level text features, and the language tags are used as input to the language transformer, and high-level language features are generated. .

[0029] Furthermore, visual feature enhancement modules such as Figure 3 As shown in the figure, it includes two parts: a language-guided visual feature enhancement module and a multimodal feature stacking module: (1) Language-guided visual feature enhancement module: for high-level visual features extracted by the visual module And high-level language features extracted by the language module , apply a linear projection layer to transform the language features Projection to visual features Update visual features among features with the same dimension Dimension, will As Query, The multi-head self-attention layer of the Transformer is used as the key and value for calculation, and then the calculation result is mapped through a linear layer and combined with the initial Add the residuals to get the high-level visual features enhanced by language guidance , and finally Re-enter the visual feature enhancement module as input and calculate again , a total of 3 cycles, to obtain the final visual features enhanced by language guidance .

[0030] (2) Multimodal feature stacking module: high-level language features After the steps of dimension transformation, dimension increase, replication, and reshape, a new language feature tensor is obtained after expansion. , the high-level visual features enhanced by language guidance With the new language feature tensor Concat is performed on the channel dimension to obtain the concatenated features. .

[0031] Furthermore, the visual language fusion module, such as Figure 4 As shown, first randomly initialize a It is used to capture the global information of multimodal fusion, further integrate the visual features and text features through the multi-head self-attention mechanism of Transformer, and then As Query, the original As Key and Value, through the multi-head self-attention mechanism Each feature vector in collects visual language fusion information, and the obtained output and the original Add the residuals to get the new Melt, then Re-use as the input of the visual language fusion module to obtain further , a total of 4 cycles, to get the final Used to capture global information for multimodal fusion.

[0032] Furthermore, the prediction head module uses the visual language fusion module As input to the prediction head, a regression module is attached to On the token, the regression module is implemented by an MLP with two hidden layers with ReLU activations and a linear output layer, outputting the ship coordinates.

[0033] Step S3 inputs the sample data in the vision-language joint ocean ship positioning data set into the model, calculates the total loss function value, performs back propagation, optimizes the connection weights through the selected optimizer and corresponding parameters, and obtains the final vision-language joint positioning ocean ship model after multiple rounds of training; Calculate the total loss function value of the model Including smooth loss , broad sense loss and loss , the total loss function of the model is The calculation formula is:

[0034] in, Weight factor used to balance these losses.

[0035] (1) Smooth loss , and its calculation formula is as follows: , in, Indicates The error between the predicted value and the true value.

[0036] (2) Broad sense loss , and its calculation formula is as follows: , in, For the general , and its calculation formula is: , in, represents the standard intersection-over-union ratio, represents the union area of ​​the predicted box and the true box, Represents the area of ​​the outer bounding box of the predicted box and the true box. The outer bounding box is the smallest rectangle that can enclose these two boxes.

[0037] (3) loss , and its calculation formula is as follows: , in, To enhance , and its calculation formula is: , in, It represents the distance between the center of the predicted box and the center of the real box. and are the width and height of the minimum bounding box, is a small constant to avoid division by zero errors.

[0038] Step S4, based on the trained vision-language joint positioning model for oceanic ships, input the ocean remote sensing ship image and the text description corresponding to the specific ship, and output the specific coordinates of the ship corresponding to the text description. The specific ship refers to the corresponding ship in the ocean remote sensing ship image.

[0039] The present invention constructs a new visual-language joint positioning marine ship dataset based on the original marine remote sensing ship related image data, and at the same time, makes full use of language information and retains the spatial information of visual features. Specifically, firstly, a new dataset construction method is designed, including three steps of data filtering, attribute extraction and expression generation to construct a new marine remote sensing ship image visual-language joint positioning dataset, and at the same time, a visual feature enhancement module is used to enhance the visual features through language guidance before the visual-language fusion module. On this basis, the language features are expanded and concat with the visual features to retain the spatial information of the visual features, and then the multi-head self-attention mechanism of Transformer is used to perform feature fusion, so as to realize the interaction and integration of cross-modal information, and a new loss calculation method is introduced to make the loss calculation more accurate. Finally, the efficiency and accuracy of the visual-language joint positioning of marine ships are improved.

[0040] Although preferred embodiments of the present application have been described, additional changes and modifications may be made to these embodiments once those skilled in the art are aware of the basic inventive concepts.

[0041] Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment as well as all changes and modifications that fall within the scope of the present application.

[0042] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A Transformer-based vision-language joint ocean ship positioning method, characterized in that: include: Step S1, collecting ocean remote sensing image data and preprocessing it to obtain an ocean ship positioning data set; Step S2, constructing a Transformer-based vision-language joint marine ship positioning model, the model includes a vision module, a language module, a visual feature enhancement module, a vision-language fusion module and a prediction head module; the vision module in step S2 takes a given marine remote sensing ship image as input, uses a backbone network to perform feature extraction to generate a two-dimensional feature map , and then use convolution to reduce The channel dimension of Flattening, using sinusoidal spatial position encoding as The complement of , and use the visual Transformer to perform global visual context reasoning, the output is High-level visual features with the same shape ; Step S3, inputting the sample data in the ocean ship positioning data set into the model, calculating the total loss function value, performing back propagation, optimizing the connection weights through the selected optimizer and model parameters, and obtaining a trained vision-language joint ocean ship positioning model; Step S4, input the ocean remote sensing ship image and the text description corresponding to the ship into the trained visual-language joint ocean ship positioning model, and output the specific coordinates of the ship corresponding to the text description.

2. According to claim 1, a Transformer-based vision-language combined ocean ship positioning method is characterized in that: The step S1 comprises: Step S11, data filtering: filtering the ocean remote sensing ship-related image data, deleting non-ship category data, irregular data and erroneous data, and obtaining image data suitable for vision-language combined ocean ship positioning; Step S12, attribute extraction; the attribute extraction determines a set of attributes applicable to the marine remote sensing ship data, extracts the attributes of the marine remote sensing ship according to different attribute extraction methods, and obtains a marine remote sensing ship attribute dictionary; Step S13, expression generation; the expression generation determines a set of text description sentence templates suitable for marine remote sensing ship data, selects appropriate sentence templates to fill in attribute information according to the marine remote sensing ship attribute dictionary, obtains the text expression of the marine remote sensing ship, saves all generated text expressions into a list, writes them into the corresponding label file, and obtains the visual-language joint marine ship positioning data set.

3. The method for marine ship positioning based on Transformer vision-language combination according to claim 1, characterized in that: The language module in step S2 includes a token embedding layer and a language transformer. For text expressions, each word is embedded into a one-hot embedding vector, each one-hot vector is converted into a language tag, and a [CLS] tag and a [SEP] tag are appended. A BERT model with 6 Transformer encoder layers containing pre-trained weights is used to extract word-level text features and sentence-level text features, and the language tags are used as inputs of the language transformer to generate high-level language features. .

4. The method for marine ship positioning based on Transformer vision-language combination according to claim 1, characterized in that: The visual feature enhancement module in step S2 includes a language-guided visual feature enhancement module and a multimodal feature stacking module; the language-guided visual feature enhancement module is used to extract the high-level visual features from the visual module. And high-level language features extracted by the language module , apply a linear projection layer to transform the language features Projection to visual features Update visual features among features with the same dimension Dimension, will As Query, The multi-head self-attention layer of the Transformer is used as the key and value for calculation, and then the calculation result is mapped through a linear layer and combined with the initial Add the residuals to get the high-level visual features enhanced by language guidance ; The multimodal feature stacking module combines high-level language features After the steps of dimension transformation, dimension increase, replication, and reshape, a new language feature tensor is obtained after expansion. , the high-level visual features enhanced by language guidance With the new language feature tensor Perform splicing on the channel dimension to obtain the spliced ​​features .

5. The method for marine ship positioning based on Transformer vision-language combination according to claim 1, characterized in that: The visual language fusion module in step S2 is used to first randomly initialize a It is used to capture the global information of multimodal fusion, further integrate the visual features and text features through the multi-head self-attention mechanism of Transformer, and then As Query, the original As Key and Value, through the multi-head self-attention mechanism Each feature vector in collects visual language fusion information, and the obtained output and the original Add the residuals to get the new .

6. The method of claim 1 for combining vision and language based on Transformer for marine ship positioning, characterized in that: The prediction head module in step S2 is used to utilize the visual language fusion module As input to the prediction head, a regression module is attached to On the token, the regression module is implemented by an MLP with two hidden layers with ReLU activations and a linear output layer, outputting the ship coordinates.

7. The method of claim 1 for combining vision and language based on Transformer for marine ship positioning, characterized in that: In step S3, the total loss function value of the model is calculated Including smooth loss , broad sense loss and loss , the total loss function of the model is The calculation formula is: ； in, Weight factor used to balance these losses.

8. A computer device comprising a memory, a processor and program instructions stored in the memory and executable by the processor, wherein the processor executes the program instructions to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images