Multi-modal fusion-based ship course obstacle monitoring method and ship collision avoidance method

Abstract

The invention discloses a ship course obstacle monitoring method and a ship collision avoidance method based on multi-modal fusion. The obstacle monitoring method comprises the following steps: step 1, obtaining a visual modal image and an infrared modal image; step 2, carrying out registration on the visible and infrared modal images; 3, fusing the visual and infrared modal images by adopting a trained RFN-Nest neural network based on a residual network structure to obtain a multi-modal fusion image; step 4, training the improved YOLOv11 network by using the multi-modal fusion image to obtain a target detection network; and 5, fusing the visual and infrared modal images to be detected into a multi-modal fusion image to be detected, inputting the multi-modal fusion image to the target detection network, and identifying to obtain a target obstacle. According to the ship collision avoidance method, an obstacle monitoring method is used for monitoring a target obstacle, the distance between the target obstacle and a ship is measured when the target obstacle is recognized, and whether collision avoidance measures are taken or not is judged based on the distance. According to the invention, the safety of ship driving is improved.

Description

Technical Field

[0001] The present invention relates to the field of ship collision monitoring, and in particular to a ship course obstacle monitoring method and a ship collision avoidance method based on multimodal fusion. Background Art

[0002] In the development of the shipping industry, preventing ship collisions has always been a key issue. When ships are sailing at sea, the complex and changeable navigation factors make it difficult to use only visible light to cope with situations such as darkness, rain and fog. Therefore, this paper adopts a method of fusing infrared and visible light modalities to assist ships in collision avoidance and provide effective auxiliary information for further decision-making. Visible light images have clear resolution and rich object textures, but the image information is easily affected by complex environmental factors. Infrared images have good environmental adaptability and the outlines of objects in the images are clear. Multimodal fusion can fuse information from multiple modalities to improve the accuracy of predictions.

[0003] In the prior art, a Chinese patent document with authorization announcement number CN119091226A discloses a multimodal recognition method for ship target types under conditions of local information loss. This invention extracts local key features from visible light and infrared modal images as graph model nodes, and then fuses the extracted information nodes using a graph neural network, effectively overcoming the problem of information loss in single visible light images under environments such as fog, rain, and snow, and improving the accuracy of classification and recognition. A Chinese patent document with authorization announcement number CN119810564A discloses a dynamic ship classification and recognition method based on multimodal quality changes. This method can effectively deal with the situation where the quality of infrared and visible light images is uneven, and improve the accuracy and robustness of ship recognition in complex scenarios. However, the above inventions either do not have image registration or image fusion, etc., and their accuracy needs to be improved. It is difficult to accurately identify obstacles when used for ship collision monitoring, so their application is limited. Summary of the Invention

[0004] The present invention provides a ship course obstacle monitoring method and a ship collision avoidance method based on multimodal fusion, so as to solve the problem that the existing ship multimodal method is difficult to apply to obstacle identification during ship collision monitoring and difficult for ships to automatically avoid collisions due to insufficient accuracy.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A ship course obstacle monitoring method and a ship collision avoidance method based on multimodal fusion, comprising the following steps:

[0007] Step 1: Acquire multiple visual modality images and multiple infrared modality images of the ship's heading. The multiple visual modality images and the multiple infrared modality images are synchronized in time and space in a one-to-one correspondence to form a training data set.

[0008] Step 2: registering the visual modality image and the infrared modality image obtained in step 1 to adjust the sizes of the visual modality image and the infrared modality image to the same size;

[0009] Step 3: Use the data set processed in step 2 as the training set to train the RFN-Nest neural network based on the residual network structure. The trained RFN-Nest neural network is used to fuse the visual modality image and infrared modality image processed in step 2 to obtain a multimodal fusion image.

[0010] Step 4: Construct a training set using the multimodal fusion image obtained in step 3, and train the improved YOLOv11 network so that the improved YOLOv11 network can identify the target obstacle from the multimodal fusion image, and the trained improved YOLOv11 network is used as the target detection network;

[0011] Step 5: Obtain a visual modal image and an infrared modal image to be detected that are synchronized in time and space with respect to the vessel's heading. Based on steps 2 and 3, fuse the visual modal image and the infrared modal image to be detected into a multimodal fused image to be detected.

[0012] Then, the multimodal fusion image to be detected is input into the target detection network obtained in step 4, and the target detection network identifies the target obstacle from the multimodal fusion image to be detected, thereby completing the monitoring of the obstacle.

[0013] In the further step 2, the SIFT algorithm is used to register the visual modality image and the infrared modality image.

[0014] In the further step 4, the RFN-Nest neural network includes an encoder network, an RFN network as a residual fusion network for extracting and fusing multi-scale deep features, and a decoder network based on nest connection.

[0015] Furthermore, the RFN-Nest neural network is trained using a two-stage training method, wherein the encoder network and the decoder network are trained in the first stage, and the weights of each layer of the encoder network and the decoder network are fixed after the first stage of training;

[0016] In the second stage, the RFN network is added to the decoder network and encoder network with fixed weights at each layer to form a complete RFN-Nest neural network for training, and the weights and parameters of each convolutional layer of the RFN network are adjusted based on the loss function during training.

[0017] In step 5, the improved YOLOv11 network is obtained by improving the C2PSA module in the YOLOv11 network to an I-C2PSA module;

[0018] Among them, the attention mechanism of the PSA module of the C2PSA module in the YOLOv11 network is changed to a variable large kernel module, thereby obtaining the I-C2PSA module.

[0019] A method for avoiding collision of ships, the process is as follows:

[0020] According to the above-mentioned ship course obstacle monitoring method based on multimodal fusion, the target obstacle on the ship course is identified;

[0021] When a target obstacle is identified, the distance between the target obstacle and the ship is measured and compared with the set safety distance. If the measured distance is less than or equal to the set safety distance, collision avoidance measures are taken.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This invention adopts deep neural network and multimodal perception technology, and proposes a ship course obstacle monitoring method and ship collision avoidance method based on multimodal fusion. By fusing visible light and infrared light images, the problem of ship docking in complex environments (such as night, rain and fog) is solved by using image registration, multimodal fusion of RFN-Nest network and target recognition technology based on improved YOLOv11 algorithm. It can accurately monitor target obstacles in low-light environments such as night, and provide more accurate decision-making auxiliary information for ship collision avoidance.

[0024] The present invention also discloses a ship collision avoidance method. This method, based on a multimodal fusion ship course obstacle monitoring method, automatically monitors target obstacles on the course. Upon identifying an obstacle, it automatically measures the distance between the target obstacle and the ship. When the measured distance is less than or equal to the safe distance, collision avoidance measures are automatically taken. This allows the ship to automatically identify and monitor obstacles on the course and automatically avoid collisions when an obstacle is identified, thereby improving the safety of the ship's navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the overall process of the method according to the embodiment of the present invention.

[0026] Figure 2 This is a network structure diagram of RFN-Nest according to an embodiment of the present invention.

[0027] Figure 3 This is a diagram of the RFN network structure of an embodiment of the present invention.

[0028] Figure 4 This is a diagram of the decoder network structure of an embodiment of the present invention.

[0029] Figure 5 This is a diagram of the improved YOLOv11 network structure according to an embodiment of the present invention.

[0030] Figure 6 2 is a schematic diagram of the C2PSA module structure in the YOLOv11 network according to an embodiment of the present invention.

[0031] Figure 7 Schematic diagram of the improved I-C2PSA module structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below with reference to the accompanying drawings and examples.

[0033] like Figure 1 As shown, this embodiment discloses a ship course obstacle monitoring method and a ship collision avoidance method based on multimodal fusion, comprising the following steps:

[0034] Step 1: Acquire multiple visual modality images and multiple infrared modality images of the ship's heading. The multiple visual modality images and the multiple infrared modality images are synchronized in time and space in a one-to-one correspondence to form a training data set.

[0035] In this example, a camera and a thermal imager simultaneously and at the same angle capture the vessel's heading, acquiring multiple visual modality images and multiple infrared modality images. Since both the camera and the thermal imager capture images simultaneously and at the same angle, the multiple visual modality images and the multiple infrared modality images acquired are synchronized in time and space. Ultimately, this example acquires a total of 707 visual modality and infrared modality images.

[0036] Step 2: Using the SIFT algorithm, register each pair of temporally and spatially synchronized visual modality images and infrared modality images obtained in step 1, so as to adjust the size of each pair of temporally and spatially synchronized visual modality images and infrared modality images to the same size.

[0037] In this embodiment, the registration process of the SIFT algorithm is as follows:

[0038] Step 2-1, detect scale space extreme values:

[0039] The scale space of an image (in this embodiment, a visual modality image or an infrared modality image) is defined as a function L(x, y, σ), which is obtained by convolving a scale-variable Gaussian kernel function G(x, y, σ) with an image representation function I(x, y), as shown in the following formula:

[0040] L(x,y,σ)=G(x,y,σ)*I(x,y) (1)

[0041] Where (x, y) is the image pixel. σ is the scale space factor. The specific value of σ is the standard deviation of the Gaussian normal distribution. Its value represents the blur level of the input image and is positively correlated with it. The function G(x, y, σ) is defined as follows:

[0042]

[0043] The Gaussian pyramid is a new image pyramid method proposed in the SIFT algorithm based on the image pyramid. A Gaussian pyramid can contain multiple groups of image pyramids, and each image pyramid contains multiple layers. The specific construction process of the Gaussian pyramid is as follows:

[0044] In step (2-1A), the initial image is magnified to twice its size as the bottom layer (i.e., the first layer) of the image pyramid of group 0, that is, the image function of group 0 is I0(x, y).

[0045] Step (2-1B) is to convolve the first layer image function with the Gaussian convolution function G(x,y,σ), that is, I0(x,y)*G(x,y,σ), to obtain the second layer image. In the SIFT algorithm, the value of σ is fixed to 1.6. Multiply σ by a scaling factor k, and use the result of I0(x,y)*G(x,y,kσ) as the result of the third layer. Then continue this operation, and the result of the last L layer is I0(x,y)*G(x,y,k (L-2) That is, in this group (group 0) of image pyramids, the sizes of the images are the same, but the images in the first layer are convolved under different scale spatial factors to form different layers.

[0046] In step (2-1C), the third-to-last image of the previous group is downsampled by a scale factor of 2 and used as the first layer of the next group of image pyramids, and then the operation in step (2-1B) is repeated.

[0047] Step (2-1D), according to the above order, the last L-th layer of the O-th group of pyramids is I O (x,y)*G(x,y,k (L-2) σ).

[0048] In order to effectively detect the positions of stable key points in the scale space, this embodiment proposes a Gaussian difference pyramid in the SIFT algorithm, which is represented by the function D(x, y, σ):

[0049] D(x,y,σ)=[G(x,y,kσ)-G(x,y,σ)]*I(x,y) (3)

[0050] For a pair of input visual modality images and infrared modality images at the same time and angle, a Gaussian difference pyramid is established between the two to search for potential scale- and rotation-invariant interest points for image recognition in different spatial scale spaces.

[0051] Step 2-2, key point positioning:

[0052] At each point of interest, we perform a Taylor second-order expansion of the scale-space function to filter out points with low contrast. We also calculate the Hessian matrix of each point of interest using the Difference of Gaussian function, removing points where the ratio of the square of the matrix's trace to the determinant is less than a threshold set between 0.6 and 0.75. These points are removed from the interest points to obtain keypoints. After this step, all keypoints in the input pair of visible and infrared modal images are found.

[0053] Step 2-3, assign key point angles:

[0054] Based on the image function L(x,y) of each pair of temporally and spatially synchronized input visual modality and infrared modality images, the gradient magnitude m(x,y) and direction θ(x,y) are calculated respectively. The calculation formula is:

[0055]

[0056] After determining the gradient direction of a feature point, we need to quantitatively analyze the distribution of pixel gradients in the vicinity of the feature point. This involves constructing a gradient direction distribution histogram, which uses angle intervals as the horizontal axis, with the value at each point corresponding to the gradient magnitude in that direction. By analyzing the histogram's morphological characteristics, the orientation corresponding to the highest point in the distribution curve is determined as the dominant orientation of the feature point. This provides information on the location, scale, and orientation of each identified key point.

[0057] Steps 2-4, key point description and matching:

[0058] After obtaining the keypoint's position, scale, and orientation, we take a 16×16 pixel interval centered on the keypoint and divide it into 16 4×4 subintervals. Within each subinterval, we calculate the gradient magnitude and gradient histogram every 45°. The 16 subblocks yield 128 calculation results, and this 1×128 vector is defined as the keypoint descriptor vector.

[0059] The Euclidean distance between the descriptor vectors of the keypoints is used to quantify the degree of keypoint matching. The two keypoints with the smallest Euclidean distance are selected as candidates. The smallest Euclidean distance is then divided by the second-smallest value. If the result is less than a pre-set threshold, the two candidate points are considered a match. In practice, this threshold is typically set between 0.6 and 0.75.

[0060] Once all candidate points are successfully matched, a pair of temporally and spatially synchronized visual and infrared modality images is successfully paired. In the dataset obtained in step 1, the visual modality has a larger field of view, encompassing the infrared modality image. A grayscale image of the registered visual modality image is obtained, and the registered visible light image is obtained by selecting a minimum rectangle encompassing all successfully matched keypoints. Repeat steps 2-1 through 2-4 to complete the registration of all visual and infrared modality images in the dataset.

[0061] Step 3: Use the data set processed in step 2 as the training set to train the RFN-Nest neural network based on the residual network structure. The trained RFN-Nest neural network is used to fuse the visual modality image and infrared modality image processed in step 2 to obtain a multimodal fusion image.

[0062] like Figure 2 As shown, in this embodiment, the RFN-Nest neural network includes an encoder network, an RFN network as a residual fusion network for extracting and fusing multi-scale deep features, and a decoder network based on nest connection.

[0063] The encoder network structure is a convolutional neural network consisting of multiple layers of convolutional layers and pooling layers.

[0064] The RFN network is a residual fusion network used to extract fused multi-scale depth features. Its specific structure is as follows: Figure 3 As shown, is the infrared image feature extracted from the mth layer in the encoder network, is the visible light image feature extracted from the mth layer in the encoder network, It is the fusion feature output by the infrared image features and visible light image features extracted in the mth layer of the encoder network after passing through the RFN network.

[0065] The decoder network structure is as follows Figure 4 As shown, The definition of is consistent with that in the RFN network. The DCB module is a module composed of two convolutional layers connected in series.

[0066] The entire RFN-Nest neural network working process is:

[0067] Step 3-1: Neural network training. The data set obtained in step 1 is used as the training data set.

[0068] In step 3-2, after the neural network training is completed, a pair of temporally and spatially synchronized visible light and infrared light images are used as input to obtain a fused image that combines the two modalities. This process is performed on each pair of temporally and spatially synchronized images in the dataset to obtain the entire fused image dataset.

[0069] In this embodiment, when the RFN-Nest neural network is trained, the image dataset obtained in step 1 is used as the training dataset.

[0070] In this embodiment, the RFN-Nest neural network is trained using a two-stage training method.

[0071] In the first stage of training, the RFN network in the RFN-Nest neural network is removed and the encoder network and decoder network are trained. The encoder network uses L auto As the loss function for training, L auto The calculation formula is:

[0072] L auto =L pixel +λL ssim (6)

[0073] Among them, L pixel L is a pixel-level loss that constrains the reconstructed image to be similar to the input image at the pixel level. ssim is the structural similarity loss, which is used to quantify the structural similarity between two images. λ is a balancing parameter.

[0074] L pixel The calculation formula is:

[0075]

[0076] L ssim The calculation formula is:

[0077] L ssim =1-SSIM(Output,Input) (8)

[0078] SSIM() stands for structural similarity, a metric that measures the similarity between two images. Output is the output image. Input is the input image.

[0079] After the first stage of training, the weights of each layer of the encoder network and the decoder network are fixed.

[0080] In the second stage of training, the RFN network is added to the decoder network and encoder network with fixed weights at each layer to form a complete RFN-Nest neural network for training. RFN Training is done for the loss function, which is calculated as:

[0081] L RFN =αL detail +L feature (9)

[0082] Among them, L detail represents the background detail preservation loss function. α is the balance parameter. L feature Enhance the loss function for target features.

[0083] Since most of the background information of the image comes from the visible light image, the loss function L detail Used to reduce the loss of background information in the fused image. The calculation formula is:

[0084] L detail =1-SSIM(Output,Input vis ) (10)

[0085] Among them, Output is the output fusion image. vis is the input visible light image.

[0086] Because infrared images contain more salient target information, the target feature enhancement loss function L feature Used to reduce the loss of salient target information in the fused image, L feature The calculation formula is:

[0087]

[0088] Among them, M is the number of multi-scale depth features, which is set to 4; w1(m) is the balance coefficient; w vis With w ir are the weight coefficients of visible light image and infrared image respectively.

[0089] The weights and biases of each convolutional layer of the RFN network are adjusted based on the loss function during training.

[0090] The 707 visual and infrared modality images processed in step 3 were fed into the trained RFN-Nest network to generate 707 multimodal fusion images. These images combine the strengths of both visible and infrared modalities, providing a more comprehensive and accurate representation of information.

[0091] Step 4: Construct a training set using the 707 multimodal fusion images obtained in step 3. Divide the 707 multimodal fusion images into a test set, a validation set, and a training set in a ratio of 8:1:1. Train the improved YOLOv11 network so that the improved YOLOv11 network can identify target obstacles from the multimodal fusion images, and use the trained improved YOLOv11 network as the target detection network.

[0092] In this embodiment, Figure 5 As shown in Figure 1, the improved YOLOv11 network is obtained by improving the C2PSA module in the YOLOv11 network to the I-C2PSA module;

[0093] Among them, the YOLOv11 network is as follows Figure 6 The attention mechanism of the PSA module of the C2PSA module shown in the figure is changed to a variable large kernel module, thereby obtaining the I-C2PSA module. The I-C2PSA module structure is as follows Figure 7 shown.

[0094] In this embodiment, the variable large kernel module provides a receptive field similar to the attention mechanism. By using depthwise convolution, depthwise dilated convolution, and 1×1 convolution, large convolution kernels can be constructed with fewer parameters and computations. For an input of dimension H×W and channel C, a convolution kernel of K×K depthwise convolution and a depthwise separable convolution with a hole is constructed. The calculation method is:

[0095] DW=(2d-1)×(2d-1) (12)

[0096]

[0097] Among them, d is the expansion rate, DW is the convolution kernel size of the depth convolution layer, and DW-D is the convolution kernel size of the depth separable convolution layer.

[0098] Image segmentation technology has been further developed with the introduction of deformable convolution, which allows flexible adjustment of the sampling grid using integer offsets to achieve free deformation. An additional convolutional layer learns these deformations from the feature map to generate an offset field. Based on the deformation information learned from the feature map, an adaptive convolution kernel can be formed. This flexible kernel shape helps to better capture and define the boundaries of the target object. The convolution layer used to calculate the offset uses the same kernel size and dilation rate as its counterpart, and for pixel values that are not on the image grid, bilinear interpolation is used for calculation. Compared with the traditional attention mechanism, the variable large kernel module significantly reduces the amount of computation and the number of parameters while maintaining high performance.

[0099] Step 5: Obtain the visual modal image to be detected and the infrared modal image to be detected that are synchronized in time and space with the ship's heading. Based on steps 2 and 3, fuse the visual modal image to be detected and the infrared modal image to be detected into a multimodal fusion image to be detected.

[0100] Then, the multimodal fusion image to be detected is input into the target detection network obtained in step 4, and the target detection network identifies the target obstacle from the multimodal fusion image to be detected.

[0101] Therefore, this embodiment can identify target obstacles in the multimodal fusion image of the ship's heading, thereby realizing automatic monitoring of obstacles while the ship is sailing and providing accurate auxiliary decision-making information for ship collision avoidance.

[0102] This embodiment also discloses a method for avoiding collision of ships, the process of which is as follows:

[0103] Step S1: In the main controller of the ship, according to the ship heading obstacle monitoring method based on multimodal fusion, the target obstacle on the ship heading is identified.

[0104] Step S2: When a target obstacle is identified, the main controller controls the laser ranging device on the ship to measure the distance between the target obstacle and the ship, and compares the measured distance with the set safety distance.

[0105] Step S3: When the measured distance is greater than the set safety distance, the main controller determines that it is in a safe state, and then repeats steps S1 and S2.

[0106] When the measured distance is less than or equal to the set safety distance, the main controller determines it as an unsafe state, controls the alarm to sound an alarm, and the ship takes collision avoidance measures.

[0107] Step S4: After the current collision avoidance measures are completed, repeat steps S1-S3.

[0108] The preferred embodiments of the present invention are described in detail above with reference to the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. Such combinations should also be regarded as the contents disclosed in this disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0109] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the scope of the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by those skilled in the art should fall within the scope of protection of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A ship course obstacle monitoring method and ship collision avoidance method based on multimodal fusion, characterized in that: The following steps are involved: Step 1: Acquire multiple visual modality images and multiple infrared modality images of the ship's heading. The multiple visual modality images and the multiple infrared modality images are synchronized in time and space in a one-to-one correspondence to form a training data set. Step 2: registering the visual modality image and the infrared modality image obtained in step 1 to adjust the sizes of the visual modality image and the infrared modality image to the same size; Step 3: Use the data set processed in step 2 as the training set to train the RFN-Nest neural network based on the residual network structure. The trained RFN-Nest neural network is used to fuse the visual modality image and infrared modality image processed in step 2 to obtain a multimodal fusion image. Step 4: Construct a training set using the multimodal fusion image obtained in step 3, and train the improved YOLOv11 network so that the improved YOLOv11 network can identify the target obstacle from the multimodal fusion image, and the trained improved YOLOv11 network is used as the target detection network; Step 5: Obtain a visual modal image and an infrared modal image to be detected that are synchronized in time and space with respect to the vessel's heading. Based on steps 2 and 3, fuse the visual modal image and the infrared modal image to be detected into a multimodal fused image to be detected. Then, the multimodal fusion image to be detected is input into the target detection network obtained in step 4, and the target detection network identifies the target obstacle from the multimodal fusion image to be detected, thereby completing the monitoring of the obstacle.

2. A ship course obstacle monitoring method and ship collision avoidance method based on multimodal fusion according to claim 1, characterized in that: In step 2, the SIFT algorithm is used to register the visual modality image and the infrared modality image.

3. The method for monitoring ship course obstacles and the method for avoiding ship collision based on multimodal fusion according to claim 1 is characterized in that: In step 4, the RFN-Nest neural network includes an encoder network, an RFN network as a residual fusion network for extracting and fusing multi-scale deep features, and a decoder network based on nest connection.

4. A ship course obstacle monitoring method and ship collision avoidance method based on multimodal fusion according to claim 3, characterized in that: The RFN-Nest neural network is trained using a two-stage training method. The encoder network and the decoder network are trained in the first stage, and the weights of each layer of the encoder network and the decoder network are fixed after the first stage of training. In the second stage, the RFN network is added to the decoder network and encoder network with fixed weights at each layer to form a complete RFN-Nest neural network for training, and the weights and parameters of each convolutional layer of the RFN network are adjusted based on the loss function during training.

5. The method for monitoring ship course obstacles and the method for avoiding ship collision based on multimodal fusion according to claim 1 is characterized in that: In step 5, the improved YOLOv11 network is obtained by improving the C2PSA module in the YOLOv11 network to an I-C2PSA module; Among them, the attention mechanism of the PSA module of the C2PSA module in the YOLOv11 network is changed to a variable large kernel module, thereby obtaining the I-C2PSA module.

6. A method for avoiding collision of ships, characterized in that: The process is as follows: According to the ship heading obstacle monitoring method based on multimodal fusion as described in any one of claims 1 to 5, the target obstacle on the ship heading is identified; When a target obstacle is identified, the distance between the target obstacle and the ship is measured and compared with the set safety distance. If the measured distance is less than or equal to the set safety distance, collision avoidance measures are taken.

Images