A method for marine ship positioning based on semantic enhancement

Abstract

The present invention discloses a method for positioning marine vessels based on semantic enhancement, which belongs to the technical field of machine learning algorithms and computer vision. The present invention collects data sets and then performs model training; constructs a marine vessel positioning model based on semantic enhancement, and the model includes: a feature encoding module, a progressive two-stage decoding module, and an information sample learning module; inputs sample data of the data set into the model, calculates the total loss function value, performs back propagation, optimizes the connection weights through a selected optimizer and corresponding parameters, and obtains the final marine vessel positioning model after multiple rounds of training; inputs text image information to be positioned, and outputs a target prediction frame indicated by a given expression. The present invention generates accurate features based on semantically perceived visual encoding, and effectively positions targets based on a progressive two-stage decoding module. By introducing an information sample learning module, the model's understanding of complex multimodal relationships is promoted, and the efficiency and accuracy of visual positioning are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of machine learning algorithms and computer vision, and in particular relates to a marine ship positioning method based on semantic enhancement. Background Art

[0002] Marine ship localization is to locate the target ship object indicated by the language expression in the image given an image containing marine ships. This process builds a bridge between human language and the visual content of the physical world, and has great potential for intelligent navigation and natural human-computer interaction. Traditional object detection methods are limited to predefined categories in the dataset, while marine ship localization combines natural language with ship images to achieve more flexible ship localization. Visual information provides an intuitive description of attributes such as shape, color, and texture, while language information provides an abstract description of objects, events, and relationships.

[0003] The main challenge of marine ship positioning is to effectively fuse and reason about multimodal information to accurately understand and respond to the ship target indicated by a given expression. Existing methods mainly use passive semantic perception to extract visual features, resulting in insufficient semantic representation within the features, and directly combining multimodal feature fusion with visual reasoning, which increases the difficulty of model training. Therefore, how to effectively perform fine-grained interactions to extract more discriminative visual features, explore more effective decoding frameworks, and ultimately achieve more efficient target positioning has become a very important task for marine ship positioning. Summary of the invention

[0004] In view of the shortcomings of the background technology, the purpose of the present invention is to propose a method for marine ship positioning based on semantic enhancement, generate accurate features based on semantically perceived visual encoding, and effectively locate the target based on a progressive two-stage decoding module. By introducing an information sample learning module, the model's understanding of complex multimodal relationships is promoted, and the efficiency and accuracy of visual positioning are improved. The method comprises the following steps:

[0005] Step S1, collect visual positioning datasets, and use Vrsbench, OPT-RSVG, DIOR-RSVG and other commonly used datasets for model training;

[0006] Step S2, constructing a visual positioning model of a marine ship positioning method based on semantic enhancement, the model comprising: a feature encoding module, a progressive two-stage decoding module and an information sample learning module;

[0007] Step S3, inputting the samples in the data set in step S1 into the visual positioning model, calculating the total loss function value, performing back propagation, optimizing the connection weights through the optimizer and corresponding parameters, and obtaining the final semantically enhanced ocean ship positioning model after multiple rounds of training;

[0008] Step S4, based on the trained semantically enhanced ocean ship positioning model, input the text image information to be positioned, and output the target prediction box indicated by the given expression.

[0009] Preferably, the feature encoding module of step S2 includes:

[0010] Step S21, visual information extraction module; the visual information extraction module samples the input visual image and uses RestNet to extract visual features ;

[0011] Step S22, language information extraction module; the language information extraction module samples the input semantic text information and uses BERT to extract semantic features ;

[0012] Step S23, dynamic semantic perception module; the dynamic semantic perception module converts the visual features And semantic features As input, it realizes global fine-grained interaction between visual and language features to generate semantically enhanced features ;

[0013] Step S24, semantics promotes the Transformer module; the semantics promotes the Transformer module to enhance the semantic features And semantic features As input, generate repeated enhancement features .

[0014] Preferably, in step S23, the dynamic semantic perception module generates a semantic enhancement feature The process is as follows:

[0015] Step Sa1, visual features and semantic features Mapping to dimension The public feature space of dynamic attention matrix The formula is as follows:

[0016]

[0017] Among them, Conv means 1×1 convolution, MLP means multi-layer perceptron, is the scale factor;

[0018] Step Sa2, using visual features and semantic features The feature vector is obtained by Msdi formula and MLP calculation , and then And the visual features after 1×1 convolution Concatenate into a sequence , the formula is as follows:

[0019] ,

[0020]

[0021] Step Sa3, use [CLS] token To guide the sequence The dimension is from 2 Down to , semantically enhanced features are obtained through dynamic text guidance , the formula is as follows:

[0022]

[0023] Preferably, the semantics in step S24 drives the Transformer module to generate repeated enhancement features The process is as follows:

[0024] Step Sb1, in order to adapt to the Transformer architecture, Flattened to ， Using Global Features and semantic enhancement features Generate significant coefficients via dot product , the global feature Through the sigmoid function Coefficient Zoom in and get visual enhancement features , the formula is as follows:

[0025] ,

[0026]

[0027] Step Sb2: The semantic push for language feature enhancement in the semantic push Transformer module uses an average strategy to obtain global language statistics. , and generate semantic enhancement features through dot product and global language statistics The important coefficient between , then, by generating the importance coefficient Using the sigmoid function To zoom in, Mapping to multi-head self-attention layer to obtain semantically enhanced queries , the specific calculation formula is as follows:

[0028] ,

[0029]

[0030] Preferably, the progressive two-stage decoding module in step S2 repeatedly enhances the features output by the encoder. and semantic features Perform association integration and output the bounding box used to predict the target object. The specific steps are as follows:

[0031] Step Sc1, the first stage decoding layer, the encoder uses and The attention mechanism outputs repeated enhanced features and semantic features , concatenate these two features into a sequence ,Will Input to Standard multimodal transformers with an additional learnable position embedding at the input of the progressive two-stage decoder , the first stage process is as follows:

[0032] ,

[0033] ,

[0034]

[0035] The above Indicates a query, Indicates the key, Represents the value, initialized to , MHA( ) is multi-head self-attention, LN( ) is layer normalization, FFN( ) represents feedforward neural network, and after initialization, multi-head attention mechanism, feedforward neural network and layer normalization are processed and output ,in, For the first ( -1) The output sequence of the layer, is the input sequence of the first layer;

[0036] Step Sc2, the second stage decoding layer, output sequence Divided into visual features and language features , and at the same time, embed the position Divided into visual position embedding and language position embedding , based on language features As query, visual features As keys and values, these divided features are then further fed into the second stage to decode the location of the target object step by step. The process is as follows:

[0037] ,

[0038] ,

[0039]

[0040] Among them, Initialize to , represents the output of the h-1th layer, when When it is 0, ;

[0041] In step Sc3, the features of each layer are input into a three-layer MLP with ReLU activation function to predict the bounding box of the target object.

[0042] Preferably, the information sample learning module in step S2 is smoothed The loss value and GIoU loss value are recorded as and ,calculate Weights of loss function and GIoU loss function , the formula is as follows:

[0043] ,

[0044]

[0045] in, is a learnable parameter used to adjust the attention paid to informative samples.

[0046] Preferably, the total loss function value of the calculation model in step S3 is ， In the second stage of the progressive two-stage decoder, the predicted bounding boxes of all layers are supervised, and the total loss between the predicted boxes of each layer and the true boxes is The calculation is as follows:

[0047]

[0048] in, is the weight function, Indicates Bounding box loss, represents the bounding box regression loss, Indicates Bounding box loss, represents the bounding box position loss, and finally The losses are summed over all bounding boxes.

[0049] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for positioning an ocean vessel based on semantic enhancement is implemented.

[0050] Compared with the prior art, the present invention proposes a method for marine ship positioning based on semantic enhancement. The method generates more accurate features based on semantically-aware visual encoding, and effectively and accurately locates the target based on progressive two-stage decoding. In addition, the method introduces information sample learning to promote the model's understanding of complex multimodal relationships. Specifically, a dynamic semantic perception gate and a semantic boost transformer are first designed to improve the semantic representation of visual features by actively perceiving semantics, thereby improving the rationality and accuracy of visual features; at the same time, based on enhanced visual features, a progressive two-stage decoder is developed to perform more accurate visual scene analysis under the guidance of language clues, ultimately achieving more efficient target positioning; on this basis, information sample learning is introduced to drive the model to adaptively prioritize information samples, thereby promoting the understanding of multimodal relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is an overall flow chart of a method for marine vessel positioning based on semantic enhancement according to the present invention;

[0052] Figure 2 A framework diagram of an ocean vessel positioning model of an ocean vessel positioning method based on semantic enhancement according to the present invention;

[0053] Figure 3 A dynamic semantic perception gate flow chart of a method for positioning a marine vessel based on semantic enhancement according to the present invention;

[0054] Figure 4 This is a diagram of the semantically driven Transformer architecture of a semantically enhanced ocean vessel positioning method of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to further clearly and completely describe the technical solutions in the embodiments of the present application. It should be noted that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0056] In order to make the invention objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application are further described in detail in conjunction with the drawings in the specification: In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the advantages of the present invention will be further illustrated by comparing the embodiments in conjunction with the drawings and specific implementation methods.

[0057] The present invention proposes a method for positioning marine vessels based on semantic enhancement. The flowchart of the method is as follows: Figure 1 As shown, the steps of this method are described in detail below:

[0058] Step S1, collect visual positioning datasets, and use Vrsbench, OPT-RSVG, DIOR-RSVG and other commonly used datasets for model training.

[0059] Step S2, constructing a visual positioning model of a marine ship positioning method based on semantic enhancement, the model includes: a feature encoding module, a progressive two-stage decoding module and an information sample learning module. The entire model framework is as follows: Figure 2 As shown;

[0060] Specifically, the feature encoding module of step S2 includes:

[0061] Step S21, visual information extraction module; the visual information extraction module samples the input visual image and uses RestNet to extract visual features ;

[0062] Step S22, language information extraction module; the language information extraction module samples the input semantic text information and uses BERT to extract semantic features ;

[0063] Step S23, dynamic semantic perception module; the dynamic semantic perception module converts the visual features And semantic features As input, it realizes global fine-grained interaction between visual and language features to generate semantically enhanced features ;

[0064] Furthermore, the dynamic semantic perception module in step S23 is composed of a global spatial dynamic interaction framework and a language-guided dynamic perception gate, and the dynamic semantic perception module generates semantic enhancement features. ,like Figure 3 As shown, the process is as follows:

[0065] Step Sa1, visual features and semantic features Mapping to dimension The public feature space of Formula generation, the formula is as follows:

[0066]

[0067] Among them, Conv means 1×1 convolution, MLP means multi-layer perceptron, is the scale factor;

[0068] Step Sa2, using visual features and semantic features pass The feature vector is obtained after the formula and MLP calculation , and then And the visual features after 1×1 convolution Concatenate into a sequence , the formula is as follows:

[0069] ,

[0070]

[0071] Step Sa3, use [CLS] token To guide the sequence The dimension is from 2 Down to , semantically enhanced features are obtained through dynamic text guidance , the formula is as follows:

[0072]

[0073] Step S24, semantically-driven Transformer module; the semantically-driven Transformer module is composed of global semantic driving and language feature-enhanced semantic driving, which transforms the semantically-enhanced feature And semantic features As input, generate repeated enhancement features ,like Figure 4 As shown;

[0074] Furthermore, the semantics in step S24 drives the Transformer module to generate repeated enhancement features The process is as follows:

[0075] Step Sb1, in order to adapt to the Transformer architecture, Flattened to ， Global semantics promotes the use of global statistical features to represent the global semantics of features. Generate corresponding important coefficients to measure global features With semantic enhancement features The similarity of each local feature in a certain degree;

[0076] Global Features and semantic enhancement features Generate significant coefficients via dot product , the global feature Through the sigmoid function Coefficient Zoom in and get visual enhancement features , the formula is as follows:

[0077] ,

[0078]

[0079] Step Sb2: semantic promotion of language feature enhancement uses an average strategy to obtain global language statistics , and generate semantic enhancement features through dot product and global language statistics The important coefficient between , then, by generating the importance coefficient Using the sigmoid function To zoom in, Mapping to multi-head self-attention layer to obtain semantically enhanced queries , the specific calculation formula is as follows:

[0080] ,

[0081]

[0082] Specifically, the progressive two-stage decoding module in step S2 is implemented by connecting the stage decoding module and the prediction head in series, and the repeated enhancement feature of the encoder output and semantic features Perform association integration and output the bounding box used to predict the target object. The specific steps are as follows:

[0083] Step Sc1, the first stage decoding layer, the encoder uses and The attention mechanism outputs repeated enhanced features and semantic features , use two linear projection layers to map them to the same dimension, then concatenate the two features into a sequence ,Will Input to In the standard multimodal transformers, a fully connected correspondence between each image patch and each word is learned, and a learnable position embedding is added to the input of the progressive two-stage decoder. , the first stage process can be expressed as:

[0084] ,

[0085] ,

[0086]

[0087] The above Indicates a query, Indicates the key, Represents the value, initialized to , MHA( ) is multi-head self-attention, LN( ) is layer normalization, FFN( ) represents feedforward neural network, and after initialization, multi-head attention mechanism, feedforward neural network and layer normalization are processed and output ,in, For the first ( -1) The output sequence of the layer, is the input sequence of the first layer, when When it is 0, ;

[0088] Step Sc2, the second stage decoding layer, output sequence Divided into visual features and language features , and at the same time, embed the position Divided into visual position embedding and language position embedding , based on language features As query, visual features As keys and values, these divided features are then further fed into the second stage to decode the location of the target object step by step. The process is as follows:

[0089] ,

[0090] ,

[0091]

[0092] Among them, Initialize to , represents the output of the h-1th layer, when When it is 0, ;

[0093] In step Sc3, the features of each layer are input into a three-layer MLP with ReLU activation function to predict the bounding box of the target object.

[0094] Specifically, the information sample learning module in step S2 directly uses the loss value to measure the learning difficulty of the sample, and smoothes The loss value and GIoU loss value are recorded as and ,function Used to generate information weights for self-discovery of information samples and calculate Weights of loss function and GIoU loss function , the formula is as follows:

[0095] ,

[0096]

[0097] in, is a learnable parameter used to adjust the attention paid to informative samples.

[0098] Step S3, input the samples in the data set in step S1 into the visual positioning model, calculate the total loss function value, perform back propagation, optimize the connection weights through the optimizer and corresponding parameters, and obtain the final semantically enhanced ocean ship positioning model after multiple rounds of training.

[0099] Specifically, the total loss function value of the calculation model in step S3 is ， In the second stage of the progressive two-stage decoder, the predicted bounding boxes of all layers are supervised, and the total loss between the predicted boxes of each layer and the true boxes is The calculation is as follows:

[0100]

[0101] in, is the weight function, Indicates Bounding box loss, represents the bounding box regression loss, Indicates Bounding box loss, represents the bounding box position loss, and finally The losses are summed over all bounding boxes.

[0102] Step S4, based on the trained semantically enhanced ocean ship positioning model, input the text image information to be positioned, and output the target prediction box indicated by the given expression.

[0103] The present invention collects visual positioning data sets, uses Vrsbench, OPT-RSVG and DIOR-RSVG series and other commonly used data sets for model training; constructs a semantically enhanced marine ship positioning model, the model includes: a feature encoding module, a progressive two-stage decoding module and an information sample learning module; inputs sample data of the data set into the model, calculates the total loss function value, performs back propagation, optimizes the connection weights through the selected optimizer and corresponding parameters, and obtains the final semantically enhanced marine ship positioning model after multiple rounds of training; based on the trained semantically enhanced marine ship positioning model, inputs the text image information to be positioned, and outputs the target prediction frame indicated by the given expression. The present invention generates more accurate features based on semantically perceived visual encoding, and effectively positions the target based on progressive two-stage decoding. By introducing information sample learning, the model's understanding of complex multimodal relationships is promoted, and the efficiency and accuracy of visual positioning are improved.

[0104] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0105] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for marine ship positioning based on semantic enhancement, characterized in that: include: Step S1, collecting a visual positioning data set and using the data set for model training; Step S2, constructing a visual positioning model of a marine ship positioning method based on semantic enhancement, the model comprising: a feature encoding module, a progressive two-stage decoding module and an information sample learning module; Wherein, the feature encoding module includes: Step S21, visual information extraction module; the visual information extraction module samples the input visual image and extracts visual features I v ; Step S22, language information extraction module; the language information extraction module samples the input semantic text information and extracts semantic features F e ; Step S23, dynamic semantic perception module; the dynamic semantic perception module converts the visual feature I v And the semantic feature F e As input, generate semantically enhanced features Step S24, semantics promotes the Transformer module; the semantics promotes the Transformer module to enhance the semantic features And the semantic feature F e As input, generate repeated enhanced features F v ; The progressive two-stage decoding module repeatedly enhances the features F output by the encoder. v and semantic features F e Perform association integration and output the bounding box used to predict the target object. The specific steps are as follows: Step Sc1, the first stage decoding layer, the encoder uses X kv and X q The attention mechanism outputs repeated enhanced features F v and semantic features F e , connect these two features into a sequence F ve , F ve Input to S fsd -layer standard multimodal transformers and adds a learnable position embedding P at the input of the progressive two-stage decoder ve , the first stage process is as follows: The above Indicates a query, Indicates the key, Represents the value, initialized to MHA() stands for multi-head self-attention, LN() stands for layer normalization, and FFN() stands for feedforward neural network. After initialization, it is processed and output by multi-head attention mechanism, feedforward neural network and layer normalization. in, is the input sequence of the (l-1)th layer, is the output sequence of the lth layer, is the intermediate output sequence of the lth layer; Step Sc2, the second stage decoding layer, output sequence is divided into visual features T v and language feature T e , and at the same time, embed the position into P ve Divided into visual position embedding P v and language position embedding P e , based on the language feature T e As a query, the visual feature T v As keys and values, these divided features are then further fed into the second stage to decode the location of the target object step by step. The process is as follows: Among them, Initialize to represents the output of the h-1th layer. When h is 0, represents the key of the h-th attention head, represents the value of the h-th attention head, represents the query of the h-th attention head, T v represents a linear transformation matrix, represents the final output of the h-th attention head, represents the temporary output of the h-th attention head; Step Sc3, input the features of each layer into a three-layer MLP with ReLU activation function to predict the bounding box of the target object; In the information sample learning module, the smoothed L1 loss value and the GIoU loss value are respectively denoted as L L1 and L GIOU , calculate the weight ω of the L1 loss function and the GIoU loss function, the formula is as follows: Among them, τ is a learnable parameter used to adjust the attention to the information sample; Step S3, inputting the samples in the data set in step S1 into the visual positioning model, calculating the total loss function value, performing back propagation, optimizing the connection weights through the optimizer and corresponding parameters, and obtaining the final semantically enhanced ocean ship positioning model after multiple rounds of training; Step S4, based on the trained semantically enhanced ocean ship positioning model, input the text image information to be positioned, and output the target prediction box indicated by the given expression.

2. The method for marine ship positioning based on semantic enhancement according to claim 1, characterized in that: In step S23, the dynamic semantic perception module generates semantic enhancement features The process is as follows: Step Sa1, visual feature I v and semantic features F e Mapping to dimension D ve The public feature space of the dynamic attention matrix M sdi The formula is as follows: Among them, Conv means 1×1 convolution, MLP means multi-layer perceptron, is the scale factor; Step Sa2, using visual feature I v and semantic features F e The feature vector is obtained by Msdi formula and MLP calculation Then And the visual feature I after 1×1 convolution v Concatenate into sequence I ve , the formula is as follows: Step Sa3, use [CLS] tokenf e To guide sequence I ve The dimension changes from 2D ve Down to D ve , semantically enhanced features are obtained through dynamic text guidance The formula is as follows:

3. The method for marine ship positioning based on semantic enhancement according to claim 1, characterized in that: The semantics in step S24 drives the Transformer module to generate repeated enhancement features F v The process is as follows: Step Sb1, Flattened to ε v , using global features and semantic enhancement features ε v Generate the important coefficients M through the dot product gsb , the global feature The coefficient M is adjusted by the sigmoid function ρ gsb Scaling is performed to obtain visual enhancement feature X kv , the formula is as follows: X kv =ρ(M gsb )e v ; Step Sb2: The semantic push for language feature enhancement in the semantic push Transformer module uses an average strategy to obtain global language statistics. And generate semantic enhancement features ε through dot product v and global language statistics The important coefficient M lsb , then, by generating the importance coefficient M lsb Use the sigmoid function ρ to scale ε v Mapped to a multi-head self-attention layer to obtain a semantically enhanced query X q , the specific calculation formula is as follows: X q =ρ(M lsb )e v 。 4. The method for marine ship positioning based on semantic enhancement according to claim 1, characterized in that: The total loss function value L of the calculation model in step S3 is that in the second stage of the progressive two-stage decoder, the predicted bounding boxes of all layers are supervised, and the total loss L between the predicted boxes of each layer and the true boxes is calculated as follows: Among them, ω() is the weight function, L represents the hth bounding box L1 loss, represents the bounding box regression loss, L represents the hth bounding box GIOU loss, represents the bounding box position loss, and finally The losses are summed over all bounding boxes.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images