Ocean ship image-text positioning method based on cross-modal interaction

Abstract

The invention belongs to the technical field of smart ocean and computer vision, and discloses a cross-modal interaction-based ocean ship image-text positioning method, which comprises the following steps of: constructing a data set; constructing a coding-decoding structure model; extracting a visual feature # imgabs0 # and a linguistic feature # imgabs1 #, and generating a visual feature # imgabs2 # and a linguistic feature # imgabs3 #; fusing the above features to generate a coordinate prior # imgabs4 # and an alignment feature # imgabs5 #, enhancing an object query # imgabs7 # by using # imgabs6 #, and interacting with a multi-modal visual feature # imgabs8 # and a language feature # imgabs9 #; calculating a loss function value, and training for multiple rounds to obtain a final model. According to the method, the single-modal features are associated with other modal features, so that the semantic understanding of the model on visual and language features is improved; coordinate prior is utilized to enhance object query, a discrimination fusion module is introduced to enhance semantic consistency of multi-modal features, and finally efficiency and accuracy of a ship image-text positioning task are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of smart ocean and computer vision technology, and specifically relates to a method for positioning ocean vessels based on cross-modal interaction. Background Art

[0002] With the support of computer vision technology, marine ship image and text positioning can identify and locate the corresponding objects in the marine ship image according to the given language description. This task not only requires the model to have superb image parsing ability and be able to understand the visual information in the image, such as the shape, color, and position of the ship; it also requires the model to have strong natural language processing capabilities and be able to parse the semantic information in the language description, such as "a large cargo ship located in the center of the picture" or "a fishing boat with a red flag hanging on the right side of the stern", so as to establish an accurate mapping relationship between vision and language. The input of the image and text positioning task is an expression and an image, and the output is one or more positioning boxes, which can accurately mark the position of the object corresponding to the language description in the image. This technology has greatly improved the efficiency and accuracy of marine environmental monitoring, and is of great significance to the fields of marine environmental monitoring, marine resource exploration, and maritime rescue.

[0003] With the rise of Transformer, the use of multimodal Transformer to construct feature representation and the use of cross-attention mechanism to achieve cross-modal reasoning have achieved excellent performance in image and text localization tasks. However, this method still has problems such as difficulty in aligning visual and language information, unclear semantics of object queries, and ignoring the rich context of the image itself in marine ship images, which reduces the positioning accuracy of the model in this scene. Therefore, how to align single-modal visual and language features, enhance multimodal understanding and cross-modal interaction, and obtain more accurate object queries has now become a key task facing marine ship image and text localization. Summary of the invention

[0004] In view of the shortcomings of the background technology, the purpose of the present invention is to propose a method for positioning marine ships based on cross-modal interaction, which associates single-modal features with other modal features, improves the model's semantic understanding of visual and language features, uses coordinate priors to enhance object queries, introduces a discriminant fusion module to strengthen the semantic consistency of multimodal features, and ultimately effectively improves the efficiency and accuracy of ship positioning tasks, including the following steps: Step S1, collecting ocean ship images and text data to construct an ocean ship image and text positioning dataset; Step S2, constructing a marine ship image and text positioning model based on cross-modal interaction, the model structure includes two parts: a visual-language encoding module and a visual-language decoding module; Among them, 1) the visual-language encoding module includes a spectrum text participatory interaction module and a global visual participatory interaction module, and the specific steps are as follows: Step S21, extracting the original visual features of the image and the original language features of the text ; Step S22: the spectrum text participatory interaction module converts the original visual features and original language features Visual features used as input to generate textual participation ; Step S23, the global visual participatory interaction module converts the original visual features and original language features Generate visually engaged linguistic features as input ; 2) The visual-language decoding module includes a coordinate prior module and a discriminant fusion module, and the specific steps are as follows: Step S24: the coordinate prior module converts the original language features Generate coordinate priors as input ; Step S25, the discrimination fusion module generates visual features of text participation , Language Features of Visual Participation and original language features Generate alignment features as input ; Step S26, using coordinate priors Enhanced object query , and the extracted multimodal visual features and language features Interaction; Step S3, input the training set in the data set into the model, calculate the total loss function value, perform back propagation, optimize the connection weights through the optimizer and corresponding parameters, and obtain the final image and text positioning model after multiple rounds of training.

[0005] Preferably, the visual-language encoding module in step S21 extracts original visual features and original language features The process is as follows: ResNet is used to extract visual features ; Use BERT to extract sentence features and word features ,Will and Connecting into language features .

[0006] Preferably, the spectrum text participatory interaction module in step S22 generates visual features of text participation The process is as follows: the original visual features Perform dimensionality reduction and reshape into features ,Will As the first layer input, in each layer of the encoder, the visual features Through this module and the original language features Fusion, the feature output of the last layer is represented as the visual feature of the text , will The visual features of the layer input are recorded as , the original language features after dimension reduction Recorded as ; The specific steps are as follows: Step Sa1, use the softmax function to realize visual features and language features Global fine-grained interaction , the and Perform matrix multiplication to obtain multimodal visual features , the calculation formula is as follows: ; ; in, ,matrix , , , The shape is The learnable parameters of is the softmax function; Step Sa2, multimodal visual features Perform a two-dimensional discrete Fourier transform and set a low-pass filter to obtain visual features , and record this operation as , the calculation formula is as follows: ; ; in, represents the fast Fourier transform, represents the inverse fast Fourier transform, Indicates the use of adaptive Gaussian smoothing filter Low pass filtering is performed, and the filter The space size matches the bandwidth Adapt to the input data, Represents a convolution operation.

[0007] Preferably, the global visual participatory interaction module in step S23 generates a language feature of visual participation The process is as follows: Use multi-head cross-modal attention mechanism (MHCA) to capture the image context conditioned on the text Then, another multi-head cross-modal attention mechanism is used to fuse the image context and language features to obtain the visually involved language features ; The calculation formula is as follows: ; ; in, Represents the original language features after dimensionality reduction , Visual features that indicate textual involvement , Represents a multi-head cross-modal attention mechanism, with input parameters , and Represents query, key, and value respectively.

[0008] Preferably, the coordinate prior module in step S24 generates a coordinate prior The process is as follows: the original language features Perform average pooling to obtain features ,use Modeling latent spatial location cues and generating coordinate priors ; The calculation formula is as follows: ; ; ; ; in, represents two fully connected layers with ReLU activation function, represents the sigmoid function, is a two-dimensional anchor point, Indicates the number of output channels, is the index along the channel dimension, It's the temperature.

[0009] Preferably, the discrimination fusion module in step S25 generates the alignment feature The process is as follows: Visual features involving text and visual involvement of language features As query and key to calculate the correlation between multimodal visual features and language features, which is then applied to the original language features To collect relevant text semantics ,right and Perform multiplication to obtain the alignment feature ; The calculation formula is as follows: ; ; in, , , are the learnable projection parameters.

[0010] Preferably, in step S26, the visual-language decoding module decodes the object query Iteratively interact with multimodal language and visual features. The specific steps are as follows: Step Sb1, at level t, first add the coordinate prior P to the object query In, get , and then As a query, As keys and values, they are input into the first multi-head cross-modal attention module for interaction to obtain text semantics , the calculation formula is as follows: ; ; in, is the layer normalization operation; Step Sb2, As a query, As keys and values, they are input into the second multi-head cross-modal attention module for interaction and then normalized to collect relevant visual semantics ; Use the feedforward network to enhance the expressive power of features and then perform normalization operations, and output features , the calculation formula is as follows: ; ; in, is a feed-forward network operation.

[0011] Preferably, the model training process in step S3: use three fully connected layers and a sigmoid function to directly regress the bounding box coordinates, assuming Query for the tth layer object The projected bounding box, represents the true annotation box, and the predicted bounding box of the last layer of the decoder is used as the final prediction; Prediction box and final The loss function formula between the true annotation boxes of the layers is as follows: ; Among them, smooth L1 loss and GIoU loss For supervised model training, hyperparameters and Used to balance these two losses during training, is the number of decoder layers.

[0012] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for positioning marine vessels based on cross-modal interaction.

[0013] Compared with the prior art, the present invention proposes a method for positioning marine ships based on cross-modal interaction, which has the following advantages and beneficial effects: 1. The spectrum text participatory interaction module designed by the present invention enhances the representation of visual features by capturing and enriching the global visual language relationship. The global visual participatory interaction module designed by the present invention interprets the input text in combination with the relevant image context to generate more descriptive language features; the encoding module links the unimodal features with the relevant global context of other modalities. This process better integrates the visual features extracted by the visual encoder with the language description generated by the text encoder, realizes effective interaction between modalities through the cross-attention mechanism, and improves the model's semantic understanding of visual and language features; 2. The present invention utilizes input text to generate coordinate priors, enhances the spatial information of object query representation, suppresses inaccurate cross-modal reasoning, and further strengthens the semantic consistency of multimodal features through the discriminant fusion module, thereby more comprehensively understanding the scene and achieving efficient decoding, ultimately effectively improving the efficiency and accuracy of image and text positioning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an overall flow chart of a method for positioning marine vessels based on cross-modal interaction proposed by the present invention; Figure 2 This is a framework diagram of the marine ship graphic positioning model proposed by the present invention; Figure 3 This is a processing flow chart of the spectrum text participatory interaction module proposed by the present invention; Figure 4 This is a processing flow chart of the global visual participatory interaction module proposed in the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to further clearly and completely describe the technical solutions in the embodiments of the present application. It should be noted that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0016] In order to make the invention objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application are further described in detail in conjunction with the drawings in the specification: In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the advantages of the present invention will be further illustrated by comparing the embodiments in conjunction with the drawings and specific implementation methods.

[0017] In an embodiment, the present invention proposes a method for positioning marine vessels based on cross-modal interaction. The flowchart of the method is as follows: Figure 1 As shown, the steps of this method are described in detail below: Step S1, collect marine ship images and text data, and use OPT-RSVG, OpenSARShip and HRSC2016 datasets to build a marine ship image and text positioning dataset.

[0018] Step S2, constructing a marine ship image and text positioning model based on cross-modal interaction, the model structure includes two parts: a visual-language encoding module and a visual-language decoding module, such as Figure 2 As shown; Among them, 1) the visual-language encoding module includes a spectrum text participatory interaction module and a global visual participatory interaction module, and the specific steps are as follows: Step S21, extracting the original visual features of the image and the original language features of the text ; Furthermore, the visual-language encoding module in step S21 extracts original visual features and original language features The process is as follows: set the input image resolution to 640×640, set the length of the input text to 40, and use ResNet to extract visual features. ; Use BERT to extract sentence features and word features ,Will and Connecting into language features ; Step S22: the spectrum text participatory interaction module converts the original visual features and original language features Visual features used as input to generate textual participation ,like Figure 3 As shown; Furthermore, the spectrum text participatory interaction module in step S22 generates visual features of text participation The process is as follows: the original visual features Perform dimensionality reduction and reshape into features ,Will As the first layer input, in each layer of the encoder, the visual features Through this module and the original language features Fusion, the feature output of the last layer is represented as the visual feature of the text , will The visual features of the layer input are recorded as , the original language features after dimension reduction Recorded as ; The specific steps are as follows: Step Sa1, use the softmax function to realize visual features and language features Global fine-grained interaction , the and Perform matrix multiplication to obtain multimodal visual features , the calculation formula is as follows: ; ; in, , It is Visual features of the layer, matrix , , , The shape is The learnable parameters of is the softmax function, It is Multimodal visual features of layers; Step Sa2, multimodal visual features Perform a two-dimensional discrete Fourier transform (FFT). The first discrete Fourier transform (FFT) obtains the frequency domain representation , using an adaptive Gaussian smoothing filter The convolution operation is performed on the frequency domain representation to obtain the convolution result, the convolution result is subjected to a second inverse Fourier transform (FFT) to obtain the time domain representation, and a low-pass filter is set on the time domain to obtain the visual features. , and record this operation as , the calculation formula is as follows: ; ; in, represents the fast Fourier transform, represents the inverse fast Fourier transform, Indicates the use of adaptive Gaussian smoothing filter Low pass filtering is performed, and the filter The space size matches the bandwidth Adapt to the input data, Represents a convolution operation.

[0019] Step S23, the global visual participatory interaction module converts the original visual features and original language features Generate visually engaged linguistic features as input ,like Figure 4 As shown; Furthermore, the global visual participatory interaction module in step S23 generates a language feature of visual participation. The process is as follows: Use multi-head cross-modal attention mechanism (MHCA) to capture the image context conditioned on the text Then, another multi-head cross-modal attention mechanism is used to fuse the image context and language features to obtain the visually involved language features ; The calculation formula is as follows: ; ; in, Represents the original language features after dimensionality reduction , Visual features that indicate textual involvement , Represents a multi-head cross-modal attention mechanism, with input parameters , and Represents query, key, and value respectively.

[0020] 2) The visual-language decoding module includes a coordinate prior module and a discriminant fusion module, and the specific steps are as follows: Step S24: the coordinate prior module converts the original language features Generate coordinate priors as input ; Furthermore, the coordinate prior module in step S24 generates a coordinate prior The process is as follows: the original language features Perform average pooling to obtain features , through the fully connected layer and activation function, the features Convert to an intermediate representation ,use Modeling potential spatial position clues and generating coordinate priors through position encoding functions , then, the coordinate prior is calculated by the sine function The values ​​of the odd positions in the coordinates are calculated by the cosine function. The values ​​of the even positions in ; The calculation formula is as follows: ; ; ; ; in, represents two fully connected layers with ReLU activation function, represents the sigmoid function, represents the position encoding function, is a two-dimensional anchor point, Indicates the number of output channels, is the index along the channel dimension, It's the temperature.

[0021] Step S25, the discrimination fusion module generates visual features of text participation , Language Features of Visual Participation and original language features Generate alignment features as input ; Furthermore, the discrimination fusion module in step S25 generates alignment features The process is as follows: Visual features involving text and visual involvement of language features As query and key to calculate the correlation between multimodal visual features and language features, which is then applied to the original language features To collect relevant text semantics ,right and Perform multiplication to obtain the alignment feature ; The calculation formula is as follows: ; ; in, is a multi-head attention mechanism, , , are the learnable projection parameters.

[0022] Step S26, using coordinate priors Enhanced object query , and the extracted multimodal visual features and language features Interaction; Furthermore, in step S26, the visual-language decoding module converts the object query Iteratively interact with multimodal language and visual features to obtain a more precise representation of the referenced object. The specific steps are as follows: Step Sb1, at level t, first add the coordinate prior P to the object query In, get , and then As a query, As keys and values, they are input into the first multi-head cross-modal attention module for interaction, collecting relevant text semantics, and comparing the output of the attention module with Add and normalize to get the final text semantics , the calculation formula is as follows: ; ; in, is the layer normalization operation; Step Sb2, As a query, As keys and values, they are input into the second multi-head cross-modal attention module for interaction and then normalized to collect relevant visual semantics ; Use the feedforward network to enhance the expressiveness of the features and then perform normalization to output the features , the calculation formula is as follows: ; ; in, is the visual feature representing the current time step t, is the cross-modal attention weight, is a feed-forward network operation.

[0023] Step S3, input the training set in the data set into the model, calculate the total loss function value, perform back propagation, optimize the connection weights through the optimizer and corresponding parameters, and obtain the final image and text positioning model after multiple rounds of training.

[0024] Specifically, the model training process in step S3 is as follows: three fully connected layers and a sigmoid function are used to directly regress the bounding box coordinates. Query for the tth layer object The projected bounding box, represents the true annotation box, and the predicted bounding box of the last layer of the decoder is used as the final prediction; Prediction box and final The loss function formula between the true annotation boxes of the layers is as follows: ; Among them, smooth L1 loss and GIoU loss For supervised model training, hyperparameters and Used to balance these two losses during training, is the number of decoder layers.

[0025] In addition, an embodiment of the present invention further proposes a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned method for positioning marine vessels based on cross-modal interaction is implemented.

[0026] The present invention collects marine ship images and text data to construct a marine ship image and text positioning data set; follows the encoding-decoding structure to construct a marine ship image and text positioning model based on cross-modal interaction; extracts visual features and language features , generating visual features for text participation and visual involvement of language features ; Fusion of the above features generates coordinate priors and Align Features ,use Enhanced object query , and with multimodal visual and language features and Interaction; input the training set in the data set into the model, calculate the loss function value of the entire model, perform back propagation, optimize the connection weights through the selected optimizer and corresponding parameters, and obtain the final model after multiple rounds of training. The present invention links single-modal features with other modal features to improve the model's semantic understanding of visual and language features; uses coordinate priors to enhance object queries, introduces a discriminant fusion module to strengthen the semantic consistency of multimodal features, and ultimately effectively improves the efficiency and accuracy of marine ship image and text positioning tasks.

[0027] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0028] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for positioning marine vessels based on cross-modal interaction, characterized in that: include: Step S1, collecting ocean ship images and text data to construct an ocean ship image and text positioning dataset; Step S2, constructing a marine ship image and text positioning model based on cross-modal interaction, the model structure includes two parts: a visual-language encoding module and a visual-language decoding module; Among them, 1) the visual-language encoding module includes a spectrum text participatory interaction module and a global visual participatory interaction module, and the specific steps are as follows: Step S21, extracting the original visual features of the image and the original language features of the text ; Step S22: the spectrum text participatory interaction module converts the original visual features and original language features Visual features used as input to generate textual participation ; Step S23, the global visual participatory interaction module converts the original visual features and original language features Generate visually engaged linguistic features as input ; 2) The visual-language decoding module includes a coordinate prior module and a discriminant fusion module, and the specific steps are as follows: Step S24: the coordinate prior module converts the original language features Generate coordinate priors as input ; Step S25, the discrimination fusion module generates visual features of text participation , Language Features of Visual Participation and original language features Generate alignment features as input ; Step S26, using coordinate priors Enhanced object query , and the extracted multimodal visual features and language features Interaction; Step S3, input the training set in the data set into the model, calculate the total loss function value, perform back propagation, optimize the connection weights through the optimizer and corresponding parameters, and obtain the final image and text positioning model after multiple rounds of training.

2. According to claim 1, a method for positioning marine vessels based on cross-modal interaction is characterized in that: The visual-language encoding module in step S21 extracts the original visual features and original language features The process is as follows: ResNet is used to extract visual features , using BERT to extract sentence features and word features ,Will and Connecting into language features .

3. According to the method for positioning marine vessels based on cross-modal interaction, the method is characterized in that: The spectrum text participatory interaction module in step S22 generates visual features of text participation The process is as follows: the original visual features Perform dimensionality reduction and reshape into features ,Will As the first layer input, in each layer of the encoder, the visual features Through this module and the original language features Fusion, the feature output of the last layer is represented as the visual feature of the text , will The visual features of the layer input are recorded as , the original language features after dimension reduction Recorded as ; The specific steps are as follows: Step Sa1, use the softmax function to realize visual features and language features Global fine-grained interaction , the and Perform matrix multiplication to obtain multimodal visual features , the calculation formula is as follows: ； ； in, ,matrix , , , The shape is The learnable parameters of is the softmax function; Step Sa2, multimodal visual features Perform a two-dimensional discrete Fourier transform and set a low-pass filter to obtain visual features , and record this operation as , the calculation formula is as follows: ； ； in, represents the fast Fourier transform, represents the inverse fast Fourier transform, Indicates the use of adaptive Gaussian smoothing filter Perform low-pass filtering, Represents a convolution operation.

4. The method for positioning marine vessels based on cross-modal interaction according to claim 1, characterized in that: The global visual participatory interaction module in step S23 generates the language features of visual participation The process is as follows: Use a multi-head cross-modal attention mechanism to capture the image context under the text condition Then, another multi-head cross-modal attention mechanism is used to fuse the image context and language features to obtain the visually involved language features ; The calculation formula is as follows: ； ； in, Represents the original language features after dimensionality reduction , Visual features that indicate textual engagement , Represents a multi-head cross-modal attention mechanism, with input parameters , and Represents query, key, and value respectively.

5. The method for positioning marine vessels based on cross-modal interaction according to claim 1, characterized in that: The coordinate prior module in step S24 generates a coordinate prior The process is as follows: the original language features Perform average pooling to obtain features ,use Modeling latent spatial location cues and generating coordinate priors ; The calculation formula is as follows: ； ； ； ； in, represents two fully connected layers with ReLU activation function, represents the sigmoid function, is a two-dimensional anchor point, Indicates the number of output channels, is the index along the channel dimension, It's the temperature.

6. The method for positioning marine vessels based on cross-modal interaction according to claim 1, characterized in that: The discriminant fusion module in step S25 generates an alignment feature The process is as follows: Visual features involving text and visual involvement of language features As query and key to calculate the correlation between multimodal visual features and language features, which is then applied to the original language features To collect relevant text semantics ,right and Perform multiplication to get the alignment feature ; The calculation formula is as follows: ； ； in, , , are learnable projection parameters.

7. The method for positioning marine vessels based on cross-modal interaction according to claim 1, characterized in that: In step S26, the visual-language decoding module decodes the object query Iteratively interact with multimodal language and visual features. The specific steps are as follows: Step Sb1, at level t, first add the coordinate prior P to the object query In, get , and then As a query, As keys and values, they are input into the first multi-head cross-modal attention module for interaction to obtain text semantics , the calculation formula is as follows: ； ； in, is the layer normalization operation; Step Sb2, As a query, As keys and values, they are fed into the second multi-head cross-modal attention module for interaction to collect relevant visual semantics , using the feedforward network to enhance the expressiveness of features and output features , the calculation formula is as follows: ； ； in, is a feed-forward network operation.

8. The method for positioning marine vessels based on cross-modal interaction according to claim 1, characterized in that: The model training process in step S3 is as follows: three fully connected layers and a sigmoid function are used to directly regress the bounding box coordinates. Query for the tth layer object The projected bounding box, represents the true annotation box, and the predicted bounding box of the last layer of the decoder is used as the final prediction; Prediction box and final The loss function formula between the true annotation boxes of the layers is as follows: ； Among them, smooth L1 loss and GIoU loss For supervised model training, hyperparameters and Used to balance these two losses during training, is the number of decoder layers.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images