A ship image-text alignment method based on weather adaptive perception

Abstract

The application discloses a ship image-text alignment method based on weather adaptive perception, and comprises the following steps: obtaining an enhanced image by performing visual optimization on a ship collected image according to a weather condition vector; inputting the weather condition vector into a weather modulation network to obtain a channel weight vector and a channel bias vector; extracting features of the enhanced image by a visual coding network to obtain image region features; extracting features of a ship description text by a text coding network to obtain word-level semantic features; modulating the image region features and the text word-level features by the channel weight vector and the channel bias vector to obtain meteorological modulation visual features and meteorological modulation word-level features; and matching the meteorological modulation visual features and the meteorological modulation word-level features to obtain cross-modal alignment text features, so that adaptive cross-modal alignment of the image region features and the text word-level semantic features can be realized in a severe weather environment.

Description

Technical Field

[0001] This invention belongs to the field of image text alignment technology, specifically relating to a method for ship image text alignment based on weather adaptive perception. Background Technology

[0002] Image-text matching technology aims to establish the correspondence between visual information in images and semantic information in text, serving as a crucial foundation for applications such as cross-modal retrieval. Applying this technology to ship identification and maritime scene analysis enables the association of ship images with textual data such as ship attributes and navigation monitoring, offering significant value for maritime surveillance, target retrieval, and maritime situational awareness.

[0003] However, the real marine environment is complex. Factors such as rain, fog, low light, strong light, and sea surface reflection can easily cause degradation problems such as reduced image contrast, blurred details, missing textures, and increased noise. This leads to a shift in image feature distribution, disrupts the stable semantic relationship between images and text, and ultimately causes a decrease in cross-modal matching accuracy and poor system robustness.

[0004] To address the aforementioned issues, existing technologies typically employ preprocessing methods such as image dehazing, deraining, denoising, and enhancement to improve image quality, followed by semantic alignment using an image-text matching model. While these methods can achieve some success in specific scenarios, they still have significant shortcomings: First, most existing image enhancement methods are designed separately for individual weather degradations, lacking comprehensive perception and dynamic adaptation capabilities for real-time weather conditions, making it difficult to adjust enhancement strategies based on continuously changing environmental conditions; second, the existing image enhancement process and image-text matching process are usually separated, with the enhancement results focusing more on visual quality improvement rather than fully considering whether the enhanced image features still maintain consistency with the text semantics.

[0005] Therefore, existing technologies still lack an image-text matching method that can combine real-time weather information to collaboratively optimize image quality restoration and cross-modal semantic alignment. Summary of the Invention

[0006] This invention provides a text alignment method for ship images based on weather adaptive perception, which solves the technical problems of unstable image quality under complex weather conditions and the disconnect between image enhancement and cross-modal matching.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] The first aspect of this invention provides a method for text alignment of ship images based on weather adaptive perception, comprising:

[0009] The system acquires weather information and ship-collected images of the detected water area; constructs a raw weather vector from the weather information and inputs it into the weather coding network to obtain a weather condition vector; inputs the weather condition vector and ship-collected images into a pre-trained image enhancement model, and performs visual optimization on the ship-collected images based on the weather condition vector to obtain an enhanced image;

[0010] Weather condition vectors are input into a weather modulation network to obtain channel weight vectors and channel bias vectors; image region features are obtained by extracting features from the enhanced image through a visual coding network; and word-level semantic features are obtained by extracting features from the ship description text through a text coding network.

[0011] By using channel weight vectors and channel bias vectors to perform channel modulation on image region features and text word-level features, meteorological modulation visual features and meteorological modulation word-level features are obtained.

[0012] The weather condition vector is transformed into a scalar weather bias term using a fully connected mapping layer; meteorological modulation visual features are used as query vectors, and meteorological modulation word-level features are used as key and value vectors; the query vector, key vector, value vector, and scalar weather bias term are input into a multi-head cross-attention mechanism to obtain cross-modal aligned text features.

[0013] Furthermore, the weather information is constructed into a raw weather vector, which is then input into the weather coding network to obtain a weather condition vector, specifically including:

[0014] The weather information includes precipitation intensity, visibility, ambient light, humidity, wind speed, and time variables; the weather information is normalized and then concatenated to form a raw weather vector, expressed by the following formula:

[0015]

[0016] In the formula, This is the original weather vector. The normalized precipitation intensity, The normalized visibility, The normalized ambient illuminance, The normalized humidity. The normalized wind speed. The normalized time variable;

[0017] The raw weather vector is input into the weather coding network, and then sequentially passes through the first fully connected layer of the weather coding network. After the activation function and the second fully connected layer, the output weather condition vector is expressed by the following formula:

[0018]

[0019] In the formula, Let the weather conditions be vectors. These are the weight parameters of the first fully connected layer. These are the weight parameters of the first fully connected layer. for Activation function These are the weight parameters for the second fully connected layer. These are the weight parameters for the second fully connected layer.

[0020] Furthermore, the weather condition vector and the ship-acquired images are input into the image enhancement model, and the ship-acquired images are visually optimized based on the weather condition vector to obtain enhanced images; specifically including:

[0021] The image enhancement model includes a shared encoder, a noise suppression branch network, a dehazing branch network, a brightness correction branch network, and a weather gating network;

[0022] The shared encoder is built based on the ResNet-34 residual structure and the U-Net coding architecture. The shared encoder includes four coding modules, each of which includes a single 3×3 downsampled convolutional layer and two residual blocks stacked in sequence.

[0023] Multi-scale ship image features are obtained by feature extraction from ship-acquired images using a shared encoder; the formula is as follows:

[0024]

[0025] In the formula, For multi-scale ship image features, It is a multi-scale ship image feature set composed of the outputs of four-layer coding modules. For shared encoders; Acquire images for ships; These are the shallow ship image features output by the first-layer encoding module. The ship image features are low-level structural features output by the second-layer encoding module. The ship image features are the mid-level semantic features output by the third-layer encoding module. Ship image features are high-level semantic features output by the fourth-layer encoding module; This is the encoding result of feature extraction from the input ship-acquired image using a shared encoder.

[0026] Multi-scale ship image features are input into noise suppression, dehazing, and brightness correction branch networks respectively to obtain noise correction features, detail enhancement features, and brightness correction features; the expression formula is:

[0027]

[0028]

[0029]

[0030] In the formula, For noise suppression branch networks, For the defogging branch network, For the brightness correction branch network, For noise correction features, Enhance features for details, This is a brightness correction feature; This means that the feature set of multi-scale ship images is input into the noise suppression branch network for noise suppression processing, i.e., noise correction features; This means that a multi-scale ship image feature set is input into a dehazing branch network for dehazing restoration, i.e., detail enhancement features; This means that the multi-scale ship image feature set is input into the brightness correction branch network for brightness correction processing, i.e., brightness correction features;

[0031] The weather condition vector is input into a weather gating network to obtain branch fusion weights. These weights are then used to fuse noise correction features, detail enhancement features, and brightness correction features to obtain an enhanced image. The formula is as follows:

[0032]

[0033]

[0034] In the formula, To enhance the image, For the branch fusion weights of the noise correction features, Branch fusion weights for enhancing details and features. The branch fusion weights are for the brightness correction features. for go through Weighted noise correction contribution characteristics for go through Weighted details enhance contribution characteristics. for go through Weighted luminance correction contribution characteristics;

[0035] Furthermore, the noise suppression branch network adopts A style residual denoising decoding model is used to suppress rain and snow noise, droplet interference, and high-frequency speckle noise; the defogging branch network employs... A style-based dehazing and restoration decoding model is used to recover contour edges and tonal contrast information under low visibility conditions; the brightness correction branch network employs... A style-based brightness correction decoding model is used to correct brightness distortion and localized exposure unevenness under low-light or high-light conditions.

[0036] Furthermore, the weather vector is input into the weather gating network to obtain the branch fusion weights, expressed by the following formula:

[0037]

[0038]

[0039] In the formula, For the branch fusion weights of the noise correction features, Branch fusion weights for enhancing details and features. The branch fusion weights are for the brightness correction features; Let the weather conditions be vectors. Here is the weight matrix of the hidden layer of the weather-gated network. For the weather-gated network hidden layer bias, for Nonlinear activation function This is the output layer weight matrix of the weather gating network. For the output layer bias of the weather gating network; For noise suppression branch weights, To restore branch weights for defogging, Weights for the brightness correction branch; for Normalization function.

[0040] Furthermore, the training process of the image enhancement model includes:

[0041] Obtain clear images of the ship and weather information under different weather conditions; convert the weather information into weather condition vectors;

[0042] A degradation process is performed on a clear ship image based on a weather condition vector to obtain a degraded image; the degraded image is then input into the image enhancement model to obtain an enhanced image;

[0043] Pixel reconstruction loss, perceptual consistency loss, structure preservation loss, and identity preservation loss are calculated based on the enhanced image and the sharpened image of the ship; the formulas are as follows:

[0044]

[0045]

[0046]

[0047]

[0048] In the formula, For pixel reconstruction loss, To perceive consistency loss, To preserve structural losses, Loss of identity To enhance the image, For clear images of ships, For the preset perceptual feature extractor, For the pre-defined perceptual feature extractor from the enhanced image High-level perception features extracted from [the data]. For the pre-defined perceptual feature extractor from clear images of ships High-level perception features extracted from them; For structural similarity operations, To enhance the image Clear images of ships The degree of similarity in brightness, contrast and structural information; For the preliminary review of ship identity feature extraction network, For ship identity feature extraction networks from enhanced images The identity feature vector extracted from it, For ship identification feature extraction networks, extract from clear ship images The identity feature vector extracted from it, The error is the squared error.

[0049] The image enhancement comprehensive loss is obtained by weighted summation of pixel reconstruction loss, perceptual consistency loss, structure preservation loss and identity preservation loss. The image enhancement comprehensive loss is used as a supervision signal to calculate the parameter gradient of the image enhancement model through backpropagation. An adaptive momentum optimizer is used to update the parameters along the gradient descent direction. The training process of the image enhancement model is repeated iteratively until the image enhancement comprehensive loss converges and outputs the trained image enhancement model.

[0050] Furthermore, the weather condition vector is input into the weather modulation network to obtain the channel weight vector and channel bias vector, specifically including:

[0051]

[0052]

[0053]

[0054] In the formula, This is the output feature vector of the hidden layer of the weather modulation network. for Nonlinear activation function Here is the weight matrix of the hidden layer of the weather modulation network. Let the weather conditions be vectors. This is the linear transformation result of the weather condition vector after being mapped by the weight matrix of the hidden layer of the weather modulation network. To adjust the bias of the hidden layer of the weather modulation network, The channel weight vector for visual features. This represents the channel bias vector for visual features. For channel weight vector and channel bias vector The modulation parameters for the visual feature channels are used to adjust the weighting and bias of each visual feature channel. Here is the visual output head weight matrix of the weather gating network. For the visual output head bias of the weather gating network, for right The result after linear mapping; This represents the channel weight vector of the text features. This is the channel bias vector for the text features. For channel weight vector and channel bias vector The text feature channel modulation parameters are used to weight and bias the various dimensions of the text features. Here is the text output header weight matrix of the weather gating network. for right The result after linear mapping This is used to offset the text output header of the weather-gated network.

[0055] Furthermore, channel modulation of image region features and text word-level features is performed using channel weight vectors and channel bias vectors to obtain meteorological modulation visual features and meteorological modulation word-level features, specifically including:

[0056]

[0057]

[0058] In the formula, For image region features, For weather-modulated visual features, For text word-level features, Meteorological modulation word-level features; The channel weight vector for visual features. The channel bias vector for visual features; This represents the channel weight vector of the text features. This is the channel bias vector for the text features. This is for element-wise multiplication.

[0059] Furthermore, a fully connected mapping layer is used to transform the weather condition vector into a scalar weather bias term, specifically including:

[0060]

[0061] In the formula, For scalar weather bias, To output the transpose of the mapped weight vector, Let the weather conditions be vectors. This is the weight matrix of the fully connected mapping layer. The bias vector of the fully connected mapping layer; for Nonlinear activation function;

[0062] Furthermore, the meteorological modulation visual features are used as the query vector, and the meteorological modulation word-level features are used as the key and value vectors. The query vector, key vector, value vector, and scalar weather bias term are input into a multi-head cross-attention mechanism to obtain cross-modal aligned text features, specifically including:

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] In the formula, For query vector, To query the vector mapping matrix, For weather-modulated visual features; For key vectors, The key vector mapping matrix, Meteorological modulation word-level features; For value vectors, A value vector mapping matrix; For matrix transpose, Key vector The transpose result, For scalar weather bias, Representing feature dimension, It is a natural exponential function. The number of meteorological modulation word-level features, An index for the visual features of weather modulation. An index for meteorological modulation word-level features; For the first The first weather modulation visual feature is related to the first Attention weights for each weather modulation word-level feature; This is a cross-modal aligned text feature.

[0069] Furthermore, a text-image alignment model is composed of a weather modulation network, a visual encoding network, a text encoding network, a fully connected mapping layer, and a multi-head cross-attention mechanism;

[0070] Global similarity is calculated based on the visual features and word-level features of weather modulation, expressed by the following formula:

[0071]

[0072]

[0073]

[0074] In the formula, For global visual features, For global text features, For weather-modulated visual features, For weather modulation word-level features, The number of meteorological modulation word-level features, An index for meteorological modulation word-level features. The number of meteorological modulated visual features; An index for the visual features of weather modulation; For global similarity; Cosine similarity function The global similarity is calculated based on global visual features and global text features;

[0075] Local similarity is calculated based on cross-modal aligned text features, meteorological modulation visual features, and meteorological modulation word-level features. The formula is as follows:

[0076]

[0077] In the formula, For local similarity; The number of visual features modulated by weather. For the first A weather-modulated visual feature, For the first Cross-modal aligned text features For the first Weather modulation visual features With the Cross-modal aligned text features The cosine similarity between two text regions is used to measure the degree of matching between image region features and their corresponding text semantics. For the summation function, The number of meteorological modulation word-level features, For the first A weather modulation word-level feature, For the first Weather modulation word-level features With the Cross-modal aligned text features Cosine similarity between them.

[0078] The weather vector is input into the weather dynamic weight generation network to obtain the global similarity weight, local similarity weight, and similarity bias term, expressed by the following formula:

[0079]

[0080]

[0081] In the formula, Let the weather conditions be vectors. The hidden layer weight matrix of the weather dynamic weight generation network. The hidden layer bias is used to generate the weather dynamic weights network. for Nonlinear activation function The output layer weight matrix of the weather dynamic weight generation network. Assign output layer bias to the weather dynamic weight generation network; The initial weights for global similarity; The initial weights for local similarity; for Normalization function; For global similarity weights, For local similarity weights, This is a similarity bias term;

[0082] The image-text similarity is obtained by weighting and summing the local and global similarities using global similarity weights, local similarity weights, and a similarity bias term; the formula is as follows:

[0083]

[0084] In the formula, Image-text similarity; For global similarity, Contribute to global matching; For local similarity, Contribution value to local matching; This is a similarity bias term used to dynamically adjust the final similarity based on weather conditions;

[0085] The alignment loss is calculated based on the image-text similarity. The alignment loss is used as a supervision signal to calculate the parameter gradient of the text-image alignment model through backpropagation. An adaptive momentum optimizer is then used to update the parameters along the gradient descent direction. The updated text-image alignment model is used for the next ship image-text alignment.

[0086] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0087] Weather information is constructed into a raw weather vector, which is then input into the weather coding network to obtain a weather condition vector. The weather condition vector is then uniformly introduced into the two stages of image enhancement and image-text matching, realizing full-link weather perception control from environmental information acquisition to cross-modal alignment. Compared with the existing scheme that only performs static preprocessing on images, it can more accurately reflect the current imaging conditions.

[0088] This invention inputs weather condition vectors and ship-acquired images into a pre-trained image enhancement model. The model can adaptively adjust the weights and attention biases of each enhancement branch based on real-time weather characteristics. It can jointly optimize the image to address issues such as fog interference, insufficient lighting, and noise degradation in ship images, effectively improving image clarity, color fidelity, and detail representation. This results in high-quality enhanced images and ensures stable imaging performance of the ship visual perception system under complex and changeable sea conditions and weather.

[0089] This invention inputs weather condition vectors into a weather modulation network to obtain channel weight vectors and channel bias vectors; extracts features from the enhanced image using a visual coding network to obtain image region features; extracts features from the ship description text using a text coding network to obtain word-level semantic features; and uses the channel weight vectors and channel bias vectors to perform channel modulation on the image region features and text word-level features to obtain meteorological modulation visual features and meteorological modulation word-level features; which can effectively enhance effective information and suppress redundant interference.

[0090] This invention utilizes a fully connected mapping layer to transform weather condition vectors into scalar weather bias terms; it uses meteorological modulation visual features as query vectors and meteorological modulation word-level features as key and value vectors; it inputs the query vector, key vector, value vector, and scalar weather bias terms into a multi-head cross-attention mechanism to obtain cross-modal aligned text features, achieving adaptive cross-modal alignment between image region features and text word-level semantic features; the model can dynamically adjust constraint weights according to weather conditions, strengthening the text semantic guidance constraint in severe weather conditions and focusing on mining visual local details in clear weather conditions, thereby effectively improving the robustness and overall stability of cross-modal feature matching. Attached Figure Description

[0091] Figure 1 This is a flowchart of the ship image text alignment method provided in Embodiment 1 of the present invention;

[0092] Figure 2 This is a flowchart of image enhancement based on weather perception provided in Embodiment 1 of the present invention;

[0093] Figure 3 This is a flowchart of weather-driven text-image matching provided in Embodiment 1 of the present invention. Detailed Implementation

[0094] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0095] Example 1

[0096] like Figure 1 As shown, this embodiment provides a method for text alignment of ship images based on weather adaptive perception, including:

[0097] The image enhancement model is constructed using a shared encoder, a noise suppression branch network, a dehazing branch network, a brightness correction branch network, and a weather gating network; the pre-training process of the image enhancement model includes:

[0098] Obtain clear images of the ship and weather information under different weather conditions; convert the weather information into weather condition vectors;

[0099] A degraded image is obtained by performing a degradation process on a clear ship image based on a weather condition vector; the degradation process includes at least one or more of the following: rain and snow noise superposition, fog and haze scattering degradation, low-light brightness attenuation, and contrast compression; the degraded image is then input into the image enhancement model to obtain an enhanced image;

[0100] Pixel reconstruction loss, perceptual consistency loss, structure preservation loss, and identity preservation loss are calculated based on the enhanced image and the sharpened image of the ship; the formulas are as follows:

[0101]

[0102]

[0103]

[0104]

[0105] In the formula, For pixel reconstruction loss, To perceive consistency loss, To preserve structural losses, Loss of identity To enhance the image, For clear images of ships, For the preset perceptual feature extractor, This is a pre-defined structural similarity operation function; This invention uses a pre-screening ship identity feature extraction network; the perceptual feature extractor and ship identity feature extraction network in this invention are existing technologies.

[0106] The image enhancement comprehensive loss is obtained by weighted summation of pixel reconstruction loss, perceptual consistency loss, structure preservation loss and identity preservation loss. The image enhancement comprehensive loss is used as a supervision signal to calculate the parameter gradient of the image enhancement model through backpropagation. An adaptive momentum optimizer is used to update the parameters along the gradient descent direction. The training process of the image enhancement model is repeated iteratively until the image enhancement comprehensive loss converges and outputs the trained image enhancement model.

[0107] The process involves acquiring weather information and images collected by vessels in the monitored waters. The vessel-collected images are raw images acquired in real-time by camera equipment, shore-based monitoring equipment, shipborne imaging equipment, or unmanned platforms under current weather conditions. The weather information is then constructed into a raw weather vector and input into the weather coding network to obtain a weather condition vector. Specifically, this includes:

[0108] The weather information includes precipitation intensity, visibility, ambient light, humidity, wind speed, and time variables; the weather information is normalized and then concatenated to form a raw weather vector, expressed by the following formula:

[0109]

[0110] In the formula, This is the original weather vector. The normalized precipitation intensity, The normalized visibility, The normalized ambient illuminance, The normalized humidity. The normalized wind speed. The normalized time variable;

[0111] The raw weather vector is input into the weather coding network, and then sequentially passes through the first fully connected layer of the weather coding network. After the activation function and the second fully connected layer, the output weather condition vector is expressed by the following formula:

[0112]

[0113] In the formula, Let the weather conditions be vectors. These are the weight parameters of the first fully connected layer. These are the weight parameters of the first fully connected layer. for Activation function These are the weight parameters for the second fully connected layer. These are the weight parameters for the second fully connected layer.

[0114] like Figure 2 As shown, weather condition vectors and ship-acquired images are input into an image enhancement model. The enhanced image is obtained by visually optimizing the ship-acquired images based on the weather condition vectors. Specifically, this includes:

[0115] The shared encoder is constructed based on the ResNet-34 residual structure and the U-Net coding architecture. The shared encoder includes four coding modules, each of which includes a single 3×3 downsampled convolutional layer and two residual blocks stacked sequentially. In this embodiment, the first-stage coding module outputs 64 channels of shallow texture features, the second-stage coding module outputs 128 channels of contour features, the third-stage coding module outputs 256 channels of structural features, and the fourth-stage coding module outputs 512 channels of high-level semantic features.

[0116] Multi-scale ship image features are obtained by feature extraction from ship-acquired images using a shared encoder; the formula is as follows:

[0117]

[0118] In the formula, For shared encoders, Features of multi-scale ship images; This represents a multi-scale ship image feature set composed of the outputs of four coding modules; Acquire images for ships; , , and These are the ship image features output by the four-layer encoding module; in this embodiment... It mainly represents edge, texture, and brightness distribution information. and It mainly characterizes the ship's outline, the structure of local components, and regional hierarchical information. It mainly represents high-level semantic appearance features related to the target identity.

[0119] Multi-scale ship image features are input into noise suppression, dehazing, and brightness correction branch networks respectively to obtain noise correction features, detail enhancement features, and brightness correction features; the expression formula is:

[0120]

[0121]

[0122]

[0123] In the formula, For noise suppression branch networks, For the defogging branch network, For the brightness correction branch network, For noise correction features, Enhance features for details, This is a brightness correction feature; To process noise suppression by inputting multi-scale ship image feature sets into a noise suppression branch network, To input the multi-scale ship image feature set into the dehazing branch network for dehazing and restoration processing, To perform brightness correction processing, a multi-scale ship image feature set is input into a brightness correction branch network.

[0124] In this embodiment, the noise suppression branch network adopts... A style residual denoising decoding model is used to suppress rain and snow noise, droplet interference, and high-frequency speckle noise; the defogging branch network employs... A style-based dehazing and restoration decoding model is used to recover contour edges and tonal contrast information under low visibility conditions; the brightness correction branch network employs... A style-based brightness correction decoding model is used to correct brightness distortion and localized exposure unevenness under low-light or high-light conditions.

[0125] The weather vector is input into the weather gating network to obtain the branch fusion weights, expressed by the following formula:

[0126]

[0127]

[0128] In the formula, Let the weather conditions be vectors. Here is the weight matrix of the hidden layer of the weather-gated network. For the weather-gated network hidden layer bias, for Nonlinear activation function This is the output layer weight matrix of the weather gating network. For the output layer bias of the weather gating network; For noise suppression branch weights, To restore branch weights for defogging, Weights for the brightness correction branch; for Normalization function; For the branch fusion weights of the noise correction features, Branch fusion weights for enhancing details and features. The branch fusion weights are for the brightness correction features;

[0129] Enhanced images are obtained by fusing noise correction features, detail enhancement features, and brightness correction features using branch fusion weights. The formula is as follows:

[0130]

[0131]

[0132] In the formula, To enhance the image, For the branch fusion weights of the noise correction features, Branch fusion weights for enhancing details and features. The branch fusion weights are for the brightness correction features; For noise correction features, Enhance features for details, This is a brightness correction feature; for go through Weighted noise correction contribution characteristics for go through Weighted details enhance contribution characteristics. for go through Weighted brightness correction contribution characteristics; when weather conditions are biased towards rain and snow, Increase; when weather conditions lean towards haze or low visibility. Increase; when weather conditions are predominantly low light or strong light. This increases the intensity of the enhancement strategy, allowing it to be dynamically adjusted in response to changes in weather conditions.

[0133] This embodiment adopts an enhancement structure of "shared encoder + multi-branch recovery decoder + weather-gated fusion" and introduces weather modulation into the feature processing of each layer, so that the image enhancement model can dynamically allocate processing intensity among noise suppression, defogging recovery and brightness correction according to the current weather conditions, thereby significantly improving the image clarity and structural integrity under complex weather conditions.

[0134] like Figure 3 As shown, the enhanced image is aligned with the ship description text, specifically including:

[0135] The weather condition vector is input into the weather modulation network to obtain the channel weight vector and channel bias vector, specifically including:

[0136]

[0137]

[0138]

[0139] In the formula, This is the output feature vector of the hidden layer of the weather modulation network. for Nonlinear activation function Here is the weight matrix of the hidden layer of the weather modulation network. Let the weather conditions be vectors. This is the linear transformation result of the weather condition vector after being mapped by the weight matrix of the hidden layer of the weather modulation network. To adjust the bias of the hidden layer of the weather modulation network, The output of the hidden layer after adding the bias term is linear; The channel weight vector for visual features. This represents the channel bias vector for visual features. For channel weight vector and channel bias vector The modulation parameters for the visual feature channels are used to adjust the weighting and bias of each visual feature channel. Here is the visual output head weight matrix of the weather gating network. For the visual output head bias of the weather gating network, For visual output head weight matrix Weather hidden feature vector The result after linear mapping; This represents the channel weight vector of the text features. This is the channel bias vector for the text features. For channel weight vector and channel bias vector The text feature channel modulation parameters are used to weight and bias the various dimensions of the text features. Here is the text output header weight matrix of the weather gating network. For the text output header weight matrix Weather hidden feature vector The result after linear mapping The text output header bias is used for the weather gating network; in this embodiment, the hidden layer of the weather modulation network uses a 32-dimensional weather condition vector. Mapped to a 64-dimensional output feature vector .

[0140] Feature extraction of image regions is achieved through a visual coding network; specifically, this includes:

[0141] Visual coding networks are preferred to use Visual Model Implementation. Specifically, the augmented image is first divided into fixed-size image blocks, and each image block is linearly embedded before being input into a multi-layer... Encoder outputs image region feature sequences To align with text features, a linear projection layer is applied after the visual encoder output to uniformly map the features onto text. Dimensional shared semantic space.

[0142] Word-level semantic features are obtained by extracting features from ship description text using a text encoding network; specifically including:

[0143] Text encoding networks preferably adopt Implementation using a pre-trained language model. Specifically, the input ship description text is segmented into words before being input. The encoder obtains the contextual semantic representation of each word or word unit, which is then mapped through a linear projection layer to the same shape as the visual features. Dimensional shared semantic space, forming word-level semantic feature sequences .

[0144] By utilizing channel weight vectors and channel bias vectors to perform channel modulation on image region features and text word-level features, meteorological modulation visual features and meteorological modulation word-level features are obtained, specifically including:

[0145]

[0146]

[0147] In the formula, For image region features, For weather-modulated visual features, For text word-level features, Meteorological modulation word-level features; The channel weight vector for visual features. The channel bias vector for visual features; This represents the channel weight vector of the text features. This is the channel bias vector for the text features. This is for element-wise multiplication.

[0148] Under adverse weather conditions, the visual channel, which is more susceptible to weather interference, is suppressed. At the same time, the response intensity of discriminative terms such as color words, structural words, and attribute words in the text is enhanced, so that the two modalities can complete a weather-constrained pre-alignment before entering the interaction stage.

[0149] A fully connected mapping layer is used to transform the weather condition vector into a scalar weather bias term.

[0150] The meteorological modulation visual features are used as query vectors, and the meteorological modulation word-level features are used as key vectors and value vectors. The query vector, key vector, value vector, and scalar weather bias term are input into a multi-head cross-attention mechanism to obtain cross-modal aligned text features.

[0151] The weather condition vector is transformed into a scalar weather bias term using a fully connected mapping layer, specifically including:

[0152]

[0153] In the formula, For scalar weather bias, To output the transpose of the mapped weight vector, Let the weather conditions be vectors. This is the weight matrix of the fully connected mapping layer. The bias vector of the fully connected mapping layer; for Nonlinear activation function.

[0154] Weather modulation visual features are used as query vectors, and weather modulation word-level features are used as key and value vectors. The query vector, key vector, value vector, and scalar weather bias term are input into a multi-head cross-attention mechanism to obtain cross-modal aligned text features, specifically including:

[0155]

[0156]

[0157]

[0158]

[0159]

[0160] In the formula, For query vector, To query the vector mapping matrix, For weather-modulated visual features; For key vectors, The key vector mapping matrix, Meteorological modulation word-level features; For value vectors, A value vector mapping matrix; For matrix transpose, Key vector The transpose result, For scalar weather bias, Representing feature dimension, It is a natural exponential function. The number of meteorological modulation word-level features, An index for the visual features of weather modulation. An index for meteorological modulation word-level features; For the first The first weather modulation visual feature is related to the first Attention weights for each weather modulation word-level feature; This is a cross-modal aligned text feature.

[0161] By introducing a weather bias term into the attention scoring, the influence of text semantics on the alignment results can be appropriately increased in low-visibility weather conditions, while the contribution of local image details to the matching results can be increased in good visual conditions, thus enabling cross-modal attention allocation to change in real time with weather conditions.

[0162] The text-image alignment model consists of a weather modulation network, a visual encoding network, a text encoding network, a fully connected mapping layer, and a multi-head cross-attention mechanism. The offline or online update process for the text-image alignment model includes:

[0163] Global similarity is calculated based on the visual features and word-level features of weather modulation, expressed by the following formula:

[0164]

[0165]

[0166]

[0167] In the formula, For global visual features, For global text features, For weather-modulated visual features, For weather modulation word-level features, The number of meteorological modulation word-level features, An index for meteorological modulation word-level features. The number of meteorological modulated visual features; An index for the visual features of weather modulation; For global similarity; Cosine similarity function; This represents the global similarity calculated based on global visual features and global text features.

[0168] Local similarity is calculated based on cross-modal aligned text features, meteorological modulation visual features, and meteorological modulation word-level features. The formula is as follows:

[0169]

[0170] In the formula, For local similarity; The number of visual features modulated by weather. For the first A weather-modulated visual feature, For the first Cross-modal aligned text features For the first Weather modulation visual features With the Cross-modal aligned text features The cosine similarity between two text regions is used to measure the degree of matching between image region features and their corresponding text semantics. For the summation function, The number of meteorological modulation word-level features, For the first A weather modulation word-level feature, For the first Weather modulation word-level features With the Cross-modal aligned text features The cosine similarity between the two text regions is used; the first term in the local similarity calculation formula represents the positive local alignment from "image region to text word", and the second term represents the negative local constraint from "text word to image region". Through this bidirectional local matching, attribute words, color words, and structural words in the text can be checked against their corresponding regions in the image, while key regions in the image can also negatively constrain the completeness and accuracy of the text description, thereby achieving mutual checking matching between image content and text content.

[0171] The weather vector is input into the weather dynamic weight generation network to obtain the global similarity weight, local similarity weight, and similarity bias term, expressed by the following formula:

[0172]

[0173]

[0174] In the formula, Let the weather conditions be vectors. The hidden layer weight matrix of the weather dynamic weight generation network. The hidden layer bias is used to generate the weather dynamic weights network. for Nonlinear activation function The output layer weight matrix of the weather dynamic weight generation network. Assign output layer bias to the weather dynamic weight generation network; The initial weights for global similarity; The initial weights for local similarity; for Normalization function; For global similarity weights, For local similarity weights, This is a similarity bias term;

[0175] The image-text similarity is obtained by weighting and summing the local and global similarities using global similarity weights, local similarity weights, and a similarity bias term; the formula is as follows:

[0176]

[0177] In the formula, Image-text similarity; For global similarity, Contribute to global matching; For local similarity, Contribution value to local matching; This is a similarity bias term used to dynamically adjust the final similarity based on weather conditions. Image-text similarity calculation no longer uses a fixed single cosine similarity, but instead dynamically allocates weights between global semantic consistency and local structural alignment based on the current weather conditions: when visual degradation is weak, the contribution of global similarity is increased; when weather conditions are poor and local structural information is more critical, the contribution of local alignment similarity is increased, so that the matching score can be updated in real time with the weather conditions.

[0178] The alignment loss is calculated based on the image-text similarity. The alignment loss is used as a supervision signal to calculate the parameter gradient of the text-image alignment model through backpropagation. An adaptive momentum optimizer is then used to update the parameters along the gradient descent direction. The updated text-image alignment model is used for the next ship image-text alignment.

[0179] This embodiment employs a joint calculation method combining global similarity and local alignment similarity, and generates fusion weights and bias terms online from weather condition vectors. This allows the similarity function to be updated in real time according to weather conditions, avoiding the problem of insufficient adaptability of fixed cosine similarity in complex weather scenarios. A local attention matrix is ​​used to match the correspondence between image regions and text words item by item, enabling key parts in the image to be bidirectionally constrained with attribute words, color words, and structural words in the text, thereby achieving mutual verification between image content and text content.

[0180] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0184] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for text alignment of ship images based on weather adaptive perception, characterized in that, include: Acquire weather information and images collected by ships in the monitored waters; Weather information is constructed into a raw weather vector, which is then input into the weather coding network to obtain a weather condition vector. Weather condition vectors and ship-acquired images are input into a pre-trained image augmentation model, and the ship-acquired images are visually optimized based on the weather condition vectors to obtain augmented images. Weather condition vectors are input into a weather modulation network to obtain channel weight vectors and channel bias vectors; image region features are obtained by extracting features from the enhanced image through a visual coding network; and word-level semantic features are obtained by extracting features from the ship description text through a text coding network. By using channel weight vectors and channel bias vectors to perform channel modulation on image region features and text word-level features, meteorological modulation visual features and meteorological modulation word-level features are obtained. The weather condition vector is transformed into a scalar weather bias term using a fully connected mapping layer; meteorological modulation visual features are used as query vectors, and meteorological modulation word-level features are used as key and value vectors; the query vector, key vector, value vector, and scalar weather bias term are input into a multi-head cross-attention mechanism to obtain cross-modal aligned text features.

2. The ship image text alignment method according to claim 1, characterized in that, Weather information is constructed into a raw weather vector, which is then input into the weather coding network to obtain a weather condition vector. Specifically, this includes: The weather information includes precipitation intensity, visibility, ambient light, humidity, wind speed, and time variables; the weather information is normalized and then concatenated to form a raw weather vector, expressed by the following formula: ； In the formula, This is the original weather vector. The normalized precipitation intensity, The normalized visibility, The normalized ambient illuminance, The normalized humidity. The normalized wind speed. The normalized time variable; The raw weather vector is input into the weather coding network, and then sequentially passes through the first fully connected layer of the weather coding network. After the activation function and the second fully connected layer, the output weather condition vector is expressed by the following formula: ； In the formula, Let the weather conditions be vectors. These are the weight parameters of the first fully connected layer. These are the weight parameters of the first fully connected layer. for Activation function These are the weight parameters for the second fully connected layer. These are the weight parameters for the second fully connected layer.

3. The ship image text alignment method according to claim 1, characterized in that, Weather condition vectors and ship-acquired images are input into an image enhancement model. The enhanced image is obtained by visually optimizing the ship-acquired images based on the weather condition vectors. Specifically, this includes: The image enhancement model includes a shared encoder, a noise suppression branch network, a dehazing branch network, a brightness correction branch network, and a weather gating network; The shared encoder is built based on the ResNet-34 residual structure and the U-Net coding architecture. The shared encoder includes four coding modules, each of which includes a single 3×3 downsampled convolutional layer and two residual blocks stacked in sequence. Multi-scale ship image features are obtained by feature extraction from ship-acquired images using a shared encoder; the formula is as follows: ； In the formula, For shared encoders, Features of multi-scale ship images; Acquire images for ships; These are the shallow ship image features output by the first-layer encoding module. The ship image features are the low-level structural features output by the second-layer encoding module. The ship image features are the mid-level semantic features output by the third-layer encoding module. Ship image features are high-level semantic features output by the fourth-layer encoding module; This represents the encoding result after feature extraction from the input ship-acquired image by the shared encoder; This represents a multi-scale ship image feature set composed of the outputs of four coding modules; Multi-scale ship image features are input into noise suppression, dehazing, and brightness correction branch networks respectively to obtain noise correction features, detail enhancement features, and brightness correction features; the expression formula is: ； ； ； In the formula, For noise suppression branch networks, For the defogging branch network, For the brightness correction branch network, For noise correction features, Enhance features for details, This is a brightness correction feature; This means that the feature set of multi-scale ship images is input into the noise suppression branch network for noise suppression processing, i.e., noise correction features; This means that a multi-scale ship image feature set is input into a dehazing branch network for dehazing restoration, i.e., detail enhancement features; This means that the multi-scale ship image feature set is input into the brightness correction branch network for brightness correction processing, i.e., brightness correction features; The weather condition vector is input into a weather gating network to obtain branch fusion weights. These weights are then used to fuse noise correction features, detail enhancement features, and brightness correction features to obtain an enhanced image. The formula is as follows: ； ； In the formula, To enhance the image, For the branch fusion weights of the noise correction features, Branch fusion weights for enhancing details and features. The branch fusion weights are for the brightness correction features; for go through Weighted noise correction contribution characteristics for go through Weighted details enhance contribution characteristics. for go through Weighted luminance correction contribution characteristics.

4. The ship image text alignment method according to claim 3, characterized in that, The weather vector is input into the weather gating network to obtain the branch fusion weights, expressed by the following formula: ； ； In the formula, Let the weather conditions be vectors. Here is the weight matrix of the hidden layer of the weather-gated network. For the weather-gated network hidden layer bias, for Nonlinear activation function This is the output layer weight matrix of the weather gating network. For the output layer bias of the weather gating network; For noise suppression branch weights, To restore branch weights for defogging, Weights for the brightness correction branch; for Normalization function.

5. The ship image text alignment method according to claim 3, characterized in that, The training process of the image enhancement model includes: Obtain clear images of the ship and weather information under different weather conditions; convert the weather information into weather condition vectors; A degradation process is performed on a clear ship image based on a weather condition vector to obtain a degraded image; the degraded image is then input into the image enhancement model to obtain an enhanced image; Pixel reconstruction loss, perceptual consistency loss, structure preservation loss, and identity preservation loss are calculated based on the enhanced image and the sharpened image of the ship; the formulas are as follows: ； ； ； ； In the formula, For pixel reconstruction loss, To perceive consistency loss, To preserve structural losses, Loss of identity To enhance the image, For clear images of ships, For the preset perceptual feature extractor, For the pre-defined perceptual feature extractor from the enhanced image High-level perception features extracted from [the data]. For the pre-defined perceptual feature extractor from clear images of ships High-level perception features extracted from them; For structural similarity operations, To enhance the image Clear images of ships The degree of similarity in brightness, contrast and structural information; For the preliminary review of ship identity feature extraction network, For ship identity feature extraction networks from enhanced images The identity feature vector extracted from it, For ship identification feature extraction networks, extract from clear ship images The identity feature vector extracted from it, The error is the squared error. The image enhancement comprehensive loss is obtained by weighted summation of pixel reconstruction loss, perceptual consistency loss, structure preservation loss and identity preservation loss. The image enhancement comprehensive loss is used as a supervision signal to calculate the parameter gradient of the image enhancement model through backpropagation. An adaptive momentum optimizer is used to update the parameters along the gradient descent direction. The training process of the image enhancement model is repeated iteratively until the image enhancement comprehensive loss converges and outputs the trained image enhancement model.

6. The ship image text alignment method according to claim 1, characterized in that, The weather condition vector is input into the weather modulation network to obtain the channel weight vector and channel bias vector, specifically including: ； ； ； In the formula, This is the output feature vector of the hidden layer of the weather modulation network. for Nonlinear activation function Here is the weight matrix of the hidden layer of the weather modulation network. Let the weather conditions be vectors. This is the linear transformation result of the weather condition vector after being mapped by the weight matrix of the hidden layer of the weather modulation network. To adjust the bias of the hidden layer of the weather modulation network; The channel weight vector for visual features. This represents the channel bias vector for visual features. For channel weight vector and channel bias vector The modulation parameters for the visual feature channels are used to adjust the weighting and bias of each visual feature channel. Here is the visual output head weight matrix of the weather gating network. For the visual output head bias of the weather gating network, for right The result after linear mapping; This represents the channel weight vector of the text features. This is the channel bias vector for the text features. For channel weight vector and channel bias vector The text feature channel modulation parameters are used to weight and bias the various dimensions of the text features. Here is the text output header weight matrix of the weather gating network. for right The result after linear mapping This is used to offset the text output header of the weather-gated network.

7. The ship image text alignment method according to claim 6, characterized in that, By utilizing channel weight vectors and channel bias vectors to perform channel modulation on image region features and text word-level features, meteorological modulation visual features and meteorological modulation word-level features are obtained, specifically including: ； ； In the formula, For image region features, For weather-modulated visual features, For text word-level features, Meteorological modulation word-level features; The channel weight vector for visual features. The channel bias vector for visual features; This represents the channel weight vector of the text features. This is the channel bias vector for the text features. This is for element-wise multiplication.

8. The ship image text alignment method according to claim 1, characterized in that, The weather condition vector is transformed into a scalar weather bias term using a fully connected mapping layer, specifically including: ； In the formula, For scalar weather bias, To output the transpose of the mapped weight vector, Let the weather conditions be vectors. This is the weight matrix of the fully connected mapping layer. The bias vector of the fully connected mapping layer; for Nonlinear activation function.

9. The ship image text alignment method according to claim 8, characterized in that, Use meteorological modulation visual features as query vectors and meteorological modulation word-level features as key and value vectors; The query vector, key vector, value vector, and scalar weather bias term are input into a multi-head cross-attention mechanism to obtain cross-modal aligned text features, specifically including: ； ； ； ； ； In the formula, For query vector, To query the vector mapping matrix, For weather-modulated visual features; For key vectors, The key vector mapping matrix, Meteorological modulation word-level features; For value vectors, A value vector mapping matrix; For matrix transpose, Key vector The transpose result, For scalar weather bias, Representing feature dimension, It is a natural exponential function. The number of meteorological modulation word-level features, An index for the visual features of weather modulation. An index for meteorological modulation word-level features; For the first The first weather modulation visual feature is related to the first Attention weights for each weather modulation word-level feature; This is a cross-modal aligned text feature.

10. The ship image text alignment method according to claim 9, characterized in that, The text-image alignment model consists of a weather modulation network, a visual encoding network, a text encoding network, a fully connected mapping layer, and a multi-head cross-attention mechanism. Global similarity is calculated based on the visual features and word-level features of weather modulation, expressed by the following formula: ； ； ； In the formula, For global visual features, For global text features, For weather-modulated visual features, For weather modulation word-level features, The number of meteorological modulation word-level features, An index for meteorological modulation word-level features. The number of meteorological modulated visual features; An index for the visual features of weather modulation; For global similarity; Cosine similarity function The global similarity is calculated based on global visual features and global text features; Local similarity is calculated based on cross-modal aligned text features, meteorological modulation visual features, and meteorological modulation word-level features. The formula is as follows: ； In the formula, For local similarity; The number of visual features modulated by weather. For the first A weather-modulated visual feature, For the first Cross-modal aligned text features For the first Weather modulation visual features With the Cross-modal aligned text features The cosine similarity between two text regions is used to measure the degree of matching between image region features and their corresponding text semantics. For the summation function, The number of meteorological modulation word-level features, For the first A weather modulation word-level feature, For the first Weather modulation word-level features With the Cross-modal aligned text features Cosine similarity between them; The weather vector is input into the weather dynamic weight generation network to obtain the global similarity weight, local similarity weight, and similarity bias term, expressed by the following formula: ； ； In the formula, Let the weather conditions be vectors. The hidden layer weight matrix of the weather dynamic weight generation network. The hidden layer bias is used to generate the weather dynamic weights network. for Nonlinear activation function The output layer weight matrix of the weather dynamic weight generation network. Assign output layer bias to the weather dynamic weight generation network; The initial weights for global similarity; The initial weights for local similarity; for Normalization function; For global similarity weights, For local similarity weights, This is a similarity bias term; The image-text similarity is obtained by weighting and summing the local and global similarities using global similarity weights, local similarity weights, and a similarity bias term; the formula is as follows: ； In the formula, Image-text similarity; For global similarity, Contribute to global matching; For local similarity, Contribution value to local matching; This is a similarity bias term used to dynamically adjust the final similarity based on weather conditions; The alignment loss is calculated based on the image-text similarity. The alignment loss is used as a supervision signal to calculate the parameter gradient of the text-image alignment model through backpropagation. An adaptive momentum optimizer is then used to update the parameters along the gradient descent direction. The updated text-image alignment model is used for the next ship image-text alignment.

Images