Surface vessel detection method, device and equipment

Abstract

The present application provides a method, device, and equipment for detecting surface vessels. The method provided by the present application includes: obtaining a group of images of the same position on the water surface, and environmental information of the position; wherein the group of images includes a visible light image, a short-wave infrared image, and a long-wave infrared image; inputting the group of images and environmental information into a pre-trained detection model, so that the detection model outputs a detection result based on the group of images and environmental information. The present application provides a method, device, and equipment for detecting surface vessels, which maps text features into modulation parameters through a cross-modal fusion module in the detection model, and modulates the image fusion features of the visible light image, the short-wave infrared image, and the long-wave infrared image. It can combine environmental information to accurately guide the multi-modal integration of visible light, short-wave infrared, and long-wave infrared, thereby effectively responding to challenges brought about by environmental changes or differences in sensor quality, and can achieve more accurate and reliable surface vessel detection.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method, device and equipment for detecting surface vessels. Background Art

[0002] Surface vessel detection is crucial in multiple areas, including maritime safety, shipping management, and environmental protection. First, in maritime safety, real-time monitoring of vessel positions helps prevent collisions and improve navigation safety. Second, in the prevention and control of illegal activities, vessel detection technology can be used to identify illegal fishing, smuggling, and maritime intrusions, assisting relevant agencies in effective oversight. Furthermore, in environmental protection, timely detection and tracking of vessels illegally discharging pollutants helps reduce marine pollution and protect marine ecosystems.

[0003] With the development of artificial intelligence and remote sensing technologies, intelligent vessel detection systems based on satellite imagery, drones, and radar are improving monitoring efficiency and providing strong support for smart ocean management. However, existing surface vessel detection technologies struggle to accurately detect vessel positions in complex environments. Summary of the Invention

[0004] In view of this, the present application provides a surface vessel detection method, apparatus and device for achieving accurate and reliable surface vessel detection in complex environments.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] A first aspect of the present application provides a surface vessel detection method, the method comprising:

[0007] Acquire a set of images of the same location on the water surface and environmental information of the location; the set of images includes a visible light image, a short-wave infrared image, and a long-wave infrared image;

[0008] Inputting the set of images and the environmental information into a detection model, so that the detection model outputs a detection result based on the set of images and the environmental information;

[0009] The detection model includes an extraction module, a feature fusion module, a text processing module, a cross-modal fusion module and a detection module; the extraction module is used to extract the first feature of the visible light image, the second feature of the short-wave infrared image and the third feature of the long-wave infrared image; the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs to obtain a first fusion feature, a second fusion feature and a third fusion feature; the text processing module is used to extract features of environmental information to obtain text features; the cross-modal fusion module is used to fuse and reduce the dimension of the first fusion feature, the second fusion feature and the third fusion feature to obtain an image fusion feature; the cross-modal fusion module is also used to map the text feature to a modulation parameter, and use the image fusion feature as the feature to be processed, use the modulation parameter to process the feature to be processed to obtain a modulation feature, and process the modulation feature based on the self-attention mechanism to obtain a target feature, and then use the target feature as the feature to be processed until the number of modulations reaches a preset number, and output the final feature after modulation; the detection module is used to perform detection based on the final feature.

[0010] A second aspect of the present application provides a surface vessel detection device, comprising an acquisition module and a processing module; wherein the acquisition module is configured to acquire a set of images of a same position on the water surface and environmental information of the position; the set of images comprises a visible light image, a short-wave infrared image, and a long-wave infrared image;

[0011] The processing module is configured to input the set of images and the environmental information into a detection model, so that the detection model outputs a detection result based on the set of images and the environmental information;

[0012] The detection model includes an extraction module, a feature fusion module, a text processing module, a cross-modal fusion module and a detection module; the extraction module is used to extract the first feature of the visible light image, the second feature of the short-wave infrared image and the third feature of the long-wave infrared image; the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs to obtain a first fusion feature, a second fusion feature and a third fusion feature; the text processing module is used to extract features of environmental information to obtain text features; the cross-modal fusion module is used to fuse and reduce the dimension of the first fusion feature, the second fusion feature and the third fusion feature to obtain an image fusion feature; the cross-modal fusion module is also used to map the text feature to a modulation parameter, and use the image fusion feature as the feature to be processed, use the modulation parameter to process the feature to be processed to obtain a modulation feature, and process the modulation feature based on the self-attention mechanism to obtain a target feature, and then use the target feature as the feature to be processed until the number of modulations reaches a preset number, and output the final feature after modulation; the detection module is used to perform detection based on the final feature.

[0013] The third aspect of the present application provides a surface vessel detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0014] The surface vessel detection method, device and equipment provided in the present application collect three types of image data and environmental information, namely visible light, short-wave infrared and long-wave infrared, and fuse the image features of different modalities in pairs, and then fuse the fused features after pairwise fusion to obtain image fusion features. Furthermore, the environmental information is extracted through a text processing module, and the environmental information is introduced as a modulation parameter in the cross-modal fusion module. The image fusion features are modulated and optimized for multiple rounds, and the image fusion features are modulated and optimized for multiple rounds to achieve deep interaction between environmental information and image information, so that information of different modalities can effectively complement each other, making the final output features more global and discriminative, thereby improving detection accuracy and enhancing the model's adaptability to complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of the surface vessel detection method provided for this application;

[0016] Figure 2 A schematic diagram of a detection model shown as an exemplary embodiment of the present application;

[0017] Figure 3 This is a schematic diagram of a cross-modal fusion module according to an exemplary embodiment of the present application;

[0018] Figure 4 A schematic diagram of the structure of the surface vessel detection device provided in this application;

[0019] Figure 5 This is a schematic diagram of a surface vessel detection device shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0020] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0021] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0023] Specific embodiments are given below to introduce the technical solutions of the present application in detail.

[0024] Figure 1 The flow chart of the surface vessel detection method provided for this application. Please refer to Figure 1 The method provided in this embodiment may include:

[0025] S101. Acquire a group of images of the same position on the water surface and environmental information of the position; the group of images includes a visible light image, a short-wave infrared image, and a long-wave infrared image.

[0026] Specifically, when a location on the water surface needs to be detected, a set of images of that location and environmental information at that location are obtained. The set of images includes visible light images, short-wave infrared images, and long-wave infrared images. It should be noted that in one embodiment, visible light images can be obtained using a high-definition camera onboard a drone, short-wave infrared images can be obtained using a short-wave infrared sensor onboard the drone, and long-wave infrared images can be obtained using a long-wave infrared sensor onboard the drone.

[0027] Furthermore, environmental information at the location can be obtained from a weather station. It should be noted that the environmental information may include data such as temperature, humidity, wind speed, and air pressure.

[0028] It should be noted that by obtaining visible light images, short-wave infrared images and long-wave infrared images at the same position on the water surface to be detected, as well as the environmental information at that position, and combining this information, cross-modal joint reasoning of the ship at that water surface position can be achieved, greatly enhancing the accuracy of the detection results.

[0029] S102: Input the set of images and the environmental information into a detection model, so that the detection model outputs a detection result based on the set of images and the environmental information.

[0030] It should be noted that the detection model is pre-trained, and the training method of the pre-trained detection model can be selected according to actual needs and is not limited in this application. For example, in one embodiment, the pre-trained detection model can be trained using a supervised learning method; in another embodiment, the pre-trained detection model can be trained using a transfer learning method.

[0031] Specifically, Figure 2 This is a schematic diagram of a detection model shown in an exemplary embodiment of the present application, please refer to Figure 2 Optionally, in a possible implementation, the detection model includes a feature extraction module, a feature fusion module, a text processing module, a text-guided cross-modal fusion module and a detection module; the feature extraction module is used to extract features from the visible light image, the short-wave infrared image and the long-wave infrared image respectively, and obtain a first feature corresponding to the visible light image, a second feature corresponding to the short-wave infrared image and a third feature corresponding to the long-wave infrared image; the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs, and obtain a first fusion feature of the first feature and the second feature, a second fusion feature of the first feature and the third feature, and a third fusion feature of the second feature and the third feature; the text processing module is used to The cross-modal fusion module is used to perform feature extraction on the information to obtain text features; the cross-modal fusion module is used to perform fusion processing and dimensionality reduction processing on the first fusion feature, the second fusion feature and the third fusion feature to obtain image fusion features; the cross-modal fusion module is also used to map the text features into modulation parameters, and use the image fusion features as features to be processed, use the modulation parameters to process the features to be processed to obtain modulation features, and process the modulation features based on the self-attention mechanism to obtain target features, and then use the target features as features to be processed until the number of modulations reaches a preset number, and output the final features after modulation; the detection module is used to perform detection based on the final features output by the cross-modal fusion module.

[0032] Specifically, the feature extraction module may include a visible light encoder, a shortwave infrared encoder and a longwave infrared encoder; the visible light encoder is used to extract features from visible light images; the shortwave infrared encoder is used to extract features from shortwave infrared images; the longwave infrared encoder is used to extract features from longwave infrared images; the visible light encoder, shortwave infrared encoder and longwave infrared encoder all adopt a visual encoder based on the CLIP model, and fine-tune and adapt each modality through a low-rank adapter.

[0033] It should be noted that, in a possible implementation, when processing visible light images , shortwave infrared images , long-wave infrared images Before feature extraction, these images can be preprocessed to meet the input requirements of the feature extraction module. Preprocessing includes adjusting the image to a fixed size (the fixed size is usually 224*224) and normalizing the image according to the mean and standard deviation provided by the CLIP model. That is, for each color channel of these images (each image's color channel includes red, green, and blue), the mean and standard deviation provided by the CLIP model are used for normalization. The normalized visible light image is recorded as I vis,pre =Ρ(I vis ), the standardized short-wave infrared image is recorded as I swir,pre =Ρ(I swir ), the standardized long-wave infrared image is recorded as I iwir,pre =Ρ(I lwir ).

[0034] Specifically, the visible light encoder, shortwave infrared encoder, and longwave infrared encoder utilize a visual encoder based on the CLIP model. This visual encoder comprises three layers: a shallow feature extraction layer for extracting local image details; a mid-level semantic feature extraction layer for extracting higher-level semantic information and identifying objects or shapes in the image; and a deep semantic and global feature extraction layer for extracting global features. Each layer contains several Transform modules, and the final Transform module in each layer introduces a low-rank adapter to fine-tune the visual encoder, allowing it to be customized for data of different modalities and enhancing its feature extraction capabilities.

[0035] Among them, in the last Transform module of each level, for the multi-head attention layer inside it, the low-rank adapter is applied to the projection matrix in the multi-head attention layer. For each projection matrix , , , are fine-tuned by the low-rank adapter according to the following formula:

[0036] ;

[0037] ;

[0038] ;

[0039] in, To query the projection matrix, the size is ; For low-rank adapter pairs The adjustment matrix for fine-tuning is of the same size as same; is the fine-tuned query projection matrix;

[0040] is the key projection matrix, size is ; For low-rank adapter pairs The adjustment matrix for fine-tuning is of the same size as same; is the key projection matrix after fine-tuning;

[0041] Is the value projection matrix, size is ; For low-rank adapter pairs The adjustment matrix for fine-tuning is of the same size as same; is the key projection matrix after fine-tuning;

[0042] 、 、 is a trainable low-rank matrix of size ; 、 、 is another trainable low-rank matrix of size ; is the dimension of the low-rank matrix in the low-rank adapter.

[0043] It should be noted that 、 、 They are used to reduce the input features of the query projection matrix, key projection matrix, and value projection matrix to the dimension r of the low-rank matrix in the low-rank adapter, respectively. 、 、 They are used to map the reduced-dimensional features in the query projection matrix, key projection matrix, and value projection matrix back to the dimensions of the input features. The dimension r of the low-rank matrix is usually set to 4 or 8, which is much smaller than and ,In this way, the number of parameters for low-rank adapter fine-tuning can be reduced, and efficient fine-tuning can be achieved.

[0044] Furthermore, in the feed-forward network layer in the last Transform module of each level, the input feature X is processed by two fully connected layers:

[0045] ;

[0046] Where X is the input feature; is the weight matrix of the first fully connected layer, and its size is × , is the feature dimension of the input features of the input feedforward network layer, is the dimension of the hidden layer; Is the activation function used to output the first fully connected layer Apply nonlinear transformations; is the weight matrix of the first fully connected layer, and its size is × , is the feature dimension of the output features of the output feedforward network layer.

[0047] It should be noted that and Is the weight matrix to be learned, through the low-rank adapter and After fine-tuning, we can get the adjusted matrix:

[0048] ;

[0049] ;

[0050] in, is the first layer weight matrix; for The adjustment matrix; is the adjusted matrix; For use in A low-rank matrix for fine-tuning, size is ; For use in Another low-rank matrix for fine-tuning, size is ; is the first layer weight matrix; for The adjustment matrix; is the adjusted matrix; For use in A low-rank matrix for fine-tuning, size is ; For use in Another low-rank matrix for fine-tuning, size is .

[0051] In summary, the visible light image is input into the visible light encoder, and the visual encoder fine-tuned by the low-rank adapter processes the input visible light image to obtain the first feature of the visible light image ; Input the shortwave infrared image into the shortwave infrared encoder, and the visual encoder fine-tuned by the low-rank adapter processes the input shortwave infrared image to obtain the second feature of the shortwave infrared image ; Input the long-wave infrared image into the long-wave infrared encoder, and the visual encoder fine-tuned by the low-rank adapter processes the input long-wave infrared image to obtain the third feature of the long-wave infrared image .

[0052] Furthermore, the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs to obtain a first fused feature, a second fused feature and a third fused feature; wherein the first fused feature is obtained by fusing the first feature and the second feature, the second fused feature is obtained by fusing the first feature and the third feature, and the third fused feature is obtained by fusing the second feature and the third feature.

[0053] Optionally, in a possible implementation, the step of fusing two features to obtain a fused feature may include:

[0054] Step 1: For the two features to be fused, take the first feature of the two features as the query and the second feature as the key and value, and fuse the two features based on the cross-attention mechanism to obtain the first cross-attention feature.

[0055] Step 2: Using the second feature of the two features as the query and the first feature as the key and value, the two features are fused based on the cross-attention mechanism to obtain a second cross-attention feature.

[0056] Step 3: Concatenate the first cross-attention feature and the second cross-attention feature to obtain a fusion feature of the two features.

[0057] It should be noted that before using the cross-attention mechanism to fuse the first, second and third features in pairs, it is necessary to generate query vectors (query vector is denoted as Q), key vectors (key vector is denoted as K) and value vectors (value vector is denoted as V) adapted to their modal characteristics for the visible light modality corresponding to the first feature, the short-wave infrared modality corresponding to the second feature, and the long-wave infrared modality corresponding to the third feature, respectively, to enhance the semantic alignment of the interaction between the modalities.

[0058] Specifically, Q, K, and V corresponding to each mode can be generated based on the following formula:

[0059] ;

[0060] in, is the input feature; is the input feature The mean of the feature dimension; is the input feature variance; is a very small constant; is a learnable scaling parameter; is a learnable offset parameter; is the normalized feature.

[0061] Furthermore, based on the above formula, the Q, K, and V of each mode are obtained as shown in the following formula:

[0062] The visible light mode corresponding to the first feature:

[0063] ;

[0064] ;

[0065] ;

[0066] in, is the query vector of the visible light modality; is the bond vector of the visible light mode; is the value vector of the visible light mode; It is the first feature of the visible light image.

[0067] The shortwave infrared mode corresponding to the second feature:

[0068] ;

[0069] ;

[0070] ;

[0071] in, is the query vector of the shortwave infrared mode; is the bond vector of the shortwave infrared mode; is the value vector of the shortwave infrared mode; It is the second characteristic of the shortwave infrared mode.

[0072] The long-wave infrared mode corresponding to the third feature:

[0073] ;

[0074] ;

[0075] ;

[0076] in, is the query vector of the long-wave infrared mode; is the bond vector of the long-wave infrared mode; is the value vector of the long-wave infrared mode; It is the first characteristic of the long-wave infrared mode.

[0077] See the previous description, with the first feature and the second feature As an example of the features to be fused, first take the first feature As a query, use the visible light modality As the query vector, the second feature As keys and values, use the SWIR modality As the key vector, As the value vector, the first feature and the second feature are fused according to the following formula to obtain the first cross-attention feature:

[0078] ;

[0079] in, is the first cross-attention feature of the first feature and the second feature; is the query vector of the visible light modality; is the bond vector of the shortwave infrared mode; is the value vector of the shortwave infrared mode; is the dimension of the value vector.

[0080] Furthermore, the second feature As a query, use As the query vector, the first feature As keys and values, use As the key vector, As the value vector, the first feature and the second feature are fused according to the following formula to obtain the second cross-attention feature:

[0081] ;

[0082] in, A second cross-attention feature for the first feature and the second feature; is the query vector of the shortwave infrared mode; is the bond vector of the visible light mode; is the value vector of the visible light mode; is the dimension of the value vector.

[0083] Furthermore, the first fusion feature of the first feature and the second feature is obtained by fusion according to the following formula:

[0084] ;

[0085] in, is the first fusion feature; is the first cross-attention feature of the first feature and the second feature; is the second cross-attention feature of the first feature and the second feature.

[0086] In summary, according to the above steps, the second fusion features of the first feature and the third feature can be obtained respectively ; The third fusion feature of the second feature and the third feature .

[0087] Furthermore, the text processing module is used to extract features from the environmental information to obtain text features.

[0088] Specifically, the text processing module extracts text features using the CLIP text encoder with frozen parameters according to the following formula:

[0089] ;

[0090] in, A text description of the input (environment information in this article); For frozen CLIP text encoder; is a text feature.

[0091] It should be noted that after the text description is input into the text processing module, the CLIP text encoder in the text processing module converts the text description into word embeddings. Its internal Transform layer generates a semantic vector representation of the text, which is then processed to obtain text features with fixed dimensions. The parameter-frozen CLIP text encoder means that the text encoder is no longer trained during the detection process. Its weights are pre-trained and will not be updated, allowing direct feature extraction from the text description.

[0092] Furthermore, the fixed dimensionality of the output text features is determined by the CLIP text encoder. A text encoder with an appropriate dimensionality can be selected based on actual needs, and this application does not limit this. For example, in one embodiment, the input text description "sea fog permeates" is output as a 512-dimensional vector, with text features including "low visibility," "fog," and "ocean."

[0093] Further, Figure 3 This is a schematic diagram of a cross-modal fusion module according to an exemplary embodiment of the present application. Figure 3The cross-modal fusion module includes a fusion dimension reduction module, which is used to perform fusion processing and dimension reduction processing on the first fusion feature, the second fusion feature and the third fusion feature to obtain an image fusion feature.

[0094] Optionally, in one embodiment, the first fusion feature, the second fusion feature, and the third fusion feature may be directly concatenated together, as shown below:

[0095] ;

[0096] Furthermore, a multi-layer perceptron (MLP) is used to reduce the dimension of the fusion features, reducing the channel dimension from 3 to down to , as follows:

[0097] ;

[0098] For further information, please refer to Figure 3 The cross-modal fusion module further includes a modulation parameter generation module, which can be constructed based on a multi-layer perceptron and is configured to map the text features into modulation parameters using a first formula; wherein the first formula is:

[0099] ;

[0100] in, is the scale parameter; is the bias parameter; is the text feature; is the first mapping function; is the second mapping function; and The dynamic weight parameters for learning are updated as the environmental characteristics change.

[0101] It should be noted that the scale parameter Used to scale image features channel by channel, bias parameters Used to adjust feature offsets channel by channel, and It is a two-layer independent multi-layer perceptron. for The trainable weight parameters of for The trainable weight parameters are continuously adjusted and Optimization can enhance the cross-modal fusion module's learning of the association between text features and modulation parameters.

[0102] Furthermore, the cross-modal fusion module includes multiple interaction modules, each interaction module includes a feature modulation module and a self-attention module. Specifically, the feature adjustment module uses the modulation parameters to process the input features to be processed (for the first feature modulation module, the features to be processed are image fusion features; for the subsequent feature modulation modules, the features to be processed are target features output by the previous interaction module) to obtain modulated features.

[0103] Specifically, the modulation parameters are used to process the features to be processed to obtain the modulation features, including:

[0104] The feature to be processed is processed according to the second formula; wherein the second formula is:

[0105] ;

[0106] Among them, the is the scale parameter; is the bias parameter; is the feature to be processed; is the modulation feature; ⊙ represents element-by-element multiplication.

[0107] Furthermore, the self-attention module processes the modulation features based on the self-attention mechanism to obtain target features.

[0108] Furthermore, after multiple modulations, when the number of modulations reaches a preset number, the final characteristics after modulation are output. Figure 2 , the detection module performs detection based on the final features output by the cross-modal fusion module.

[0109] Specifically, through multiple modulations of multiple interactive modules, feature information in the modulation feature can be enhanced, thereby enhancing the accuracy of the detection result detected by the detection module.

[0110] The specific number of modulation times of the modulation feature is set according to actual needs and is not limited in this application.

[0111] Among them, when the detection module detects the final features and obtains the detection results, it will make a judgment based on its own experience database or preset database. The own experience database or preset database can be set by the operator according to actual needs, and is not limited in this application.

[0112] This solution has at least the following advantages:

[0113] (1) Improve detection accuracy.

[0114] By synergistically utilizing visible light, short-wave infrared, and long-wave infrared image data, the limitations of a single sensor are avoided, allowing the detection system to maintain high accuracy in various environments (such as lighting changes, weather influences, etc.).

[0115] In addition, a feature fusion module is used to fuse image features of different modalities pairwise, and then the fused features after pairwise fusion are fused. A step-by-step fusion method is used to perform layered processing on features of different modalities, which can extract richer multimodal information and improve the overall feature expression capability.

[0116] (2) Introduce environmental information to enhance detection robustness.

[0117] The text processing module is used to extract features of environmental information (such as meteorological conditions, temperature and humidity, wind speed, etc.), and the environmental information is introduced as a modulation parameter in the cross-modal fusion module. Multiple rounds of modulation and optimization are performed on the image fusion features to achieve deep interaction between environmental information and image information. This allows information from different modalities to effectively complement each other, making the final output features more global and discriminative, thereby improving detection accuracy and enhancing the model's adaptability to complex scenarios.

[0118] The surface vessel detection method provided in this embodiment collects three types of image data and environmental information, namely visible light, short-wave infrared and long-wave infrared, and performs pairwise fusion of image features of different modalities, and then fuses the fused features after pairwise fusion to obtain image fusion features. Furthermore, feature extraction of environmental information is performed through a text processing module, and environmental information is introduced as a modulation parameter in a cross-modal fusion module. Multiple rounds of modulation and optimization are performed on the image fusion features, and multiple rounds of modulation and optimization are performed on the image fusion features to achieve deep interaction between environmental information and image information, so that information of different modalities can effectively complement each other, making the final output features more global and discriminative, thereby improving detection accuracy and enhancing the model's adaptability to complex scenarios.

[0119] Optionally, in a possible implementation, the feature extraction module is further used to extract features of multiple scales for an image corresponding to any one modality among visible light images, short-wave infrared images, and long-wave infrared images based on the features of the image corresponding to the modality, and learn the importance of each scale feature, so as to determine the fusion ratio of the features of the multiple scales based on the importance of each scale, and fuse the features of the multiple scales according to the fusion ratio to obtain the features corresponding to the image.

[0120] Specifically, for images of different modalities, the feature extraction module can use a multi-scale convolutional neural network to extract features of different scales of the modality; for example, based on the multi-scale convolutional neural network, convolution kernels of different sizes are used (convolution kernels of different sizes can include 3×3, 5×5 and 7×7 convolution kernels) to capture different details in the image corresponding to the modality, and obtain small-scale features, medium-scale features and large-scale features; then, the attention mechanism can be used as a learning mechanism to evaluate the importance of the obtained small-scale features, medium-scale features and large-scale features; further, after obtaining the importance of each scale, the feature extraction module calculates the fusion ratio of the features of each scale in the features corresponding to the image, and according to the calculated fusion ratio, the features of multiple scales are weightedly averaged and fused, so that the fused features corresponding to the image corresponding to the modality can contain comprehensive information from different scales, thereby more comprehensively representing the hierarchy and details of the image.

[0121] Optionally, in a possible implementation, the fusing the first fusion feature, the second fusion feature, and the third fusion feature includes:

[0122] Step 1: Calculate a first correlation coefficient between the first fusion feature and the second fusion feature, a second correlation coefficient between the first fusion feature and the third fusion feature, and a third correlation coefficient between the second fusion feature and the third fusion feature.

[0123] Step 2: Determine a first weighting coefficient of the first fusion feature, a second weighting coefficient of the second fusion feature, and a third weighting coefficient of the third fusion feature based on the first correlation coefficient, the second correlation coefficient, and the third correlation coefficient.

[0124] Step 3: Use the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient to fuse the first fusion feature, the second fusion feature, and the third fusion feature.

[0125] It should be noted that when the first fusion feature, the second fusion feature and the third fusion feature are fused, each fusion feature is first dimensionality reduced to reduce the channel dimension of each fusion feature to , the first fusion feature after dimensionality reduction is recorded as , the second fusion feature after dimensionality reduction is recorded as , the third fusion feature after dimensionality reduction is recorded as .

[0126] Furthermore, a first correlation coefficient between the first fusion feature and the second fusion feature, a second correlation coefficient between the first fusion feature and the third fusion feature, and a third correlation coefficient between the second fusion feature and the third fusion feature are calculated according to the following formula:

[0127] ;

[0128] ;

[0129] ;

[0130] in, is the first correlation coefficient between the first fusion feature and the second fusion feature; is the second correlation coefficient between the first fusion feature and the third fusion feature; is the third correlation coefficient between the second fusion feature and the third fusion feature; is the first fusion feature after dimensionality reduction; is the second fusion feature after dimensionality reduction; is the third fusion feature after dimensionality reduction; is the dimension after dimensionality reduction.

[0131] Furthermore, a first weighting coefficient, a second weighting coefficient, and a third weighting coefficient are calculated based on the first correlation coefficient, the second correlation coefficient, and the third correlation coefficient according to the following formula:

[0132] ;

[0133] ;

[0134] ;

[0135] in, is the first weighting coefficient; is the second weighting coefficient; is the third weighting coefficient; + .

[0136] Furthermore, the first fusion feature, the second fusion feature, and the third fusion feature are fused according to the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient according to the following formula:

[0137] ;

[0138] in, A feature for fusing the first fusion feature, the second fusion feature, and the third fusion feature; is the first fusion feature after dimensionality reduction; is the second fusion feature after dimensionality reduction; It is the third fusion feature after dimensionality reduction.

[0139] Furthermore, the fused Perform dimensionality reduction and reduce the dimension of the fused feature from 3 to will be In this way, while reducing the amount of calculation and retaining key information, it can also adapt to the input requirements in subsequent steps. Specifically, the following formula can be used to Perform dimensionality reduction:

[0140] ;

[0141] in, For The feature vector for dimensionality reduction; A feature for fusing the first fusion feature, the second fusion feature, and the third fusion feature; It consists of two fully connected layers and a nonlinear activation function.

[0142] The surface vessel detection method provided in this embodiment first performs dimensionality reduction on the first, second, and third fused features, then uses the reduced-dimensional features to calculate the correlation coefficient between each pair of features to quantify the information association strength between different modal combinations. Based on the correlation coefficient between each pair of features, a first weighting coefficient, a second weighting coefficient, and a third weighting coefficient are generated to further enhance the contribution of key modalities and suppress unnecessary redundant information. The first, second, and third fused features are then weighted-fused based on the first, second, and third weighting coefficients. Finally, the weighted-fused features are subjected to a second dimensionality reduction. This significantly reduces computational complexity, and the features obtained after the fusion process have both strong discriminability and high robustness. This method is applicable to complex scenarios where multimodal data exhibit significant differences or where some modal information is missing, effectively improving the efficiency and accuracy of surface vessel detection in complex environments.

[0143] Corresponding to the aforementioned embodiment of a surface vessel detection method, the present application also provides an embodiment of a surface vessel detection device.

[0144] Figure 4 This is a schematic diagram of the structure of the surface vessel detection device provided in this application. Figure 4 The device provided in this embodiment includes an acquisition module 410 and a processing module 420; wherein the acquisition module 410 is used to acquire a set of images of the same position on the water surface and environmental information of the position; the set of images includes a visible light image, a short-wave infrared image, and a long-wave infrared image;

[0145] The processing module 420 is configured to input the set of images and the environmental information into a detection model, so that the detection model outputs a detection result based on the set of images and the environmental information;

[0146] The detection model includes an extraction module, a feature fusion module, a text processing module, a cross-modal fusion module and a detection module; the extraction module is used to extract the first feature of the visible light image, the second feature of the short-wave infrared image and the third feature of the long-wave infrared image; the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs to obtain a first fusion feature, a second fusion feature and a third fusion feature; the text processing module is used to extract features of environmental information to obtain text features; the cross-modal fusion module is used to fuse and reduce the dimension of the first fusion feature, the second fusion feature and the third fusion feature to obtain an image fusion feature; the cross-modal fusion module is also used to map the text feature to a modulation parameter, and use the image fusion feature as the feature to be processed, use the modulation parameter to process the feature to be processed to obtain a modulation feature, and process the modulation feature based on the self-attention mechanism to obtain a target feature, and then use the target feature as the feature to be processed until the number of modulations reaches a preset number, and output the final feature after modulation; the detection module is used to perform detection based on the final feature.

[0147] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0148] Figure 5 This is a schematic diagram of a surface vessel detection device shown in an exemplary embodiment of the present application. Figure 5 The present application also provides a surface vessel detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0149] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0150] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0151] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A surface vessel detection method, characterized in that: The method comprises: Acquire a set of images of the same location on the water surface and environmental information of the location; the set of images includes a visible light image, a short-wave infrared image, and a long-wave infrared image; Inputting the set of images and the environmental information into a detection model, so that the detection model outputs a detection result based on the set of images and the environmental information; The detection model includes an extraction module, a feature fusion module, a text processing module, a cross-modal fusion module and a detection module; the extraction module is used to extract the first feature of the visible light image, the second feature of the short-wave infrared image and the third feature of the long-wave infrared image; the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs to obtain a first fusion feature, a second fusion feature and a third fusion feature; the text processing module is used to extract features of environmental information to obtain text features; the cross-modal fusion module is used to fuse and reduce the dimension of the first fusion feature, the second fusion feature and the third fusion feature to obtain an image fusion feature; the cross-modal fusion module is also used to map the text feature to a modulation parameter, and use the image fusion feature as the feature to be processed, use the modulation parameter to process the feature to be processed to obtain a modulation feature, and process the modulation feature based on the self-attention mechanism to obtain a target feature, and then use the target feature as the feature to be processed until the number of modulations reaches a preset number, and output the final feature after modulation; the detection module is used to perform detection based on the final feature.

2. The method according to claim 1, characterized in that Mapping the text features into modulation parameters includes: Based on a multi-layer perceptron, the text features are mapped to modulation parameters using a first formula; wherein the first formula is: ； in, is the scale parameter; is the bias parameter; is the text feature; is the first mapping function; is the second mapping function; and The dynamic weight parameters for learning are updated as the environmental characteristics change.

3. The method according to claim 1 or 2, characterized in that The method of processing the feature to be processed by using the modulation parameter to obtain the modulation feature includes: The feature to be processed is processed according to the second formula; wherein the second formula is: ； described is the scale parameter; is the bias parameter; is the feature to be processed; is the modulation feature; ⊙ represents element-by-element multiplication.

4. The method according to claim 1, wherein The first feature, the second feature, and the third feature are fused in pairs to obtain a first fused feature, a second fused feature, and a third fused feature, including: For the two features to be fused, the first feature of the two features is used as the query, and the second feature is used as the key and value, and the two features are fused based on the cross-attention mechanism to obtain a first cross-attention feature; Taking the second feature of the two features as a query and the first feature as a key and a value, the two features are fused based on a cross-attention mechanism to obtain a second cross-attention feature; The first cross-attention feature and the second cross-attention feature are concatenated to obtain a fusion feature of the two features.

5. The method according to claim 1, wherein The fusing process of the first fusion feature, the second fusion feature, and the third fusion feature includes: Calculating a first correlation coefficient between the first fused feature and the second fused feature, a second correlation coefficient between the first fused feature and the third fused feature, and a third correlation coefficient between the second fused feature and the third fused feature; determining a first weighting coefficient of a first fused feature, a second weighting coefficient of a second fused feature, and a third weighting coefficient of a third fused feature based on the first correlation coefficient, the second correlation coefficient, and the third correlation coefficient; The first fusion feature, the second fusion feature, and the third fusion feature are fused using the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient.

6. The method according to claim 1, characterized in that The fusing process of the first fusion feature, the second fusion feature, and the third fusion feature includes: Calculating a first semantic similarity between the text feature and the first fused feature, a second semantic similarity between the text feature and the second fused feature, and a third semantic similarity between the text feature and the third fused feature; Determining a first weighting coefficient of a first fused feature, a second weighting coefficient of a second fused feature, and a third weighting coefficient of a third fused feature according to the first semantic similarity, the second semantic similarity, and the third semantic similarity; The first fusion feature, the second fusion feature, and the third fusion feature are fused using the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient.

7. The method according to claim 1, characterized in that The extraction module is configured to extract features at multiple scales from an image corresponding to any one of the modalities of visible light, shortwave infrared, and longwave infrared images based on the features of the image corresponding to the modality, learn the importance of the features at each scale, determine a fusion ratio of the features at multiple scales based on the importance of each scale, and fuse the features at multiple scales according to the fusion ratio to obtain the features corresponding to the image.

8. The method according to claim 1, characterized in that The extraction module includes a visible light encoder, a shortwave infrared encoder and a longwave infrared encoder; the visible light encoder is used to extract features from visible light images; the shortwave infrared encoder is used to extract features from shortwave infrared images; the longwave infrared encoder is used to extract features from longwave infrared images; the visible light encoder, shortwave infrared encoder and longwave infrared encoder all adopt a visual encoder based on the CLIP model, and are fine-tuned to adapt to each modality through a low-rank adapter.

9. A surface vessel detection device, characterized in that: The device includes an acquisition module and a processing module; wherein, The acquisition module is used to acquire a group of images of the same position on the water surface and environmental information of the position; the group of images includes visible light images, short-wave infrared images and long-wave infrared images; The processing module is configured to input the set of images and the environmental information into a detection model, so that the detection model outputs a detection result based on the set of images and the environmental information; Among them, the detection model includes an extraction module, a feature fusion module, a text processing module, a cross-modal fusion module and a detection module; the extraction module is used to extract the first feature of the visible light image, the second feature of the short-wave infrared image and the third feature of the long-wave infrared image; the feature fusion module is used to fuse the first feature, the second feature and the third feature in pairs to obtain a first fusion feature, a second fusion feature and a third fusion feature; the text processing module is used to extract features of environmental information to obtain text features; the cross-modal fusion module is used to fuse and reduce the dimension of the first fusion feature, the second fusion feature and the third fusion feature to obtain an image fusion feature; the cross-modal fusion module is also used to map the text feature to a modulation parameter, and use the image fusion feature as the feature to be processed, use the modulation parameter to process the feature to be processed to obtain a modulation feature, and process the modulation feature based on the self-attention mechanism to obtain a target feature, and then use the target feature as the feature to be processed until the number of modulations reaches a preset number and outputs the final feature after modulation; the detection module is used to perform detection based on the final feature.

10. A surface vessel detection device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the method according to any one of claims 1 to 8 are implemented when the processor executes the program.

Images