A deep-sea polymetallic nodule image segmentation method based on multi-modal data fusion

Abstract

The application belongs to the technical field of image processing, and particularly relates to a deep-sea polymetallic nodule image segmentation method based on multi-modal data fusion. The method integrates the features of photoacoustic modal images, combines an X-type multi-modal deep-sea polymetallic nodule segmentation model, fully excavates the synergistic advantages of both, and compensates for the shortcomings of single modal data. By introducing a photoacoustic fusion module and photoacoustic prediction fusion based on the degree of optical image degradation, the intermediate features and prediction results of the two modalities are dynamically fused to ensure stable nodule segmentation effect in various deep-sea environments. Through photoacoustic consistency loss, the consistency of the target region in the multi-modal prediction map is ensured, and accurate deep-sea polymetallic nodule region segmentation is achieved.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to a method for image segmentation of deep-sea polymetallic nodules based on multimodal data fusion. Background Technology

[0002] Image segmentation of deep-sea polymetallic nodules is a crucial task in deep-sea resource assessment. Currently, segmentation methods for deep-sea polymetallic nodules typically rely on single-modal data (such as sonar or optical images). Sonar images offer superior detection capabilities in deep-sea environments, especially in low-visibility or turbid waters, but suffer from poor detail rendering. Optical images provide rich surface details, but their quality degrades significantly in the harsh and unpredictable deep-sea environment. Existing nodule segmentation methods struggle to overcome the limitations of both modalities simultaneously. Therefore, combining photoacoustic modal images for fusion can fully leverage their synergistic advantages, thereby improving the performance of deep-sea polymetallic nodule image segmentation. Summary of the Invention

[0003] This application proposes a deep-sea polymetallic nodule image segmentation method based on multimodal data fusion. This method effectively improves the segmentation performance of deep-sea nodule regions by integrating features from photoacoustic modal images. The technical solution is as follows: A method for image segmentation of deep-sea polymetallic nodules based on multimodal data fusion includes the following steps: S1. Acquire image; S2. Image registration; S3. Construct a dataset based on the acquired photoacoustic images; S4. Construct an X-shaped multimodal deep-sea polymetallic nodule segmentation model; S5. Construct a loss function and train the model constructed in step S4 using the dataset constructed in step S3 to minimize the loss function; S6. Using the model trained in step S5, input the deep-sea polymetallic nodule image to be segmented, and automatically segment out the polymetallic nodule region.

[0004] Preferably, in step S2, during the image registration stage, the optical image and the acoustic image are registered to obtain photoacoustic image pairs with consistent resolution and spatial-temporal uniformity.

[0005] Preferably, the X-type multimodal deep-sea polymetallic nodule segmentation model includes a photoacoustic dual-path encoder, a photoacoustic fusion module, and a photoacoustic dual-path decoder; The photoacoustic dual-path encoder consists of an optical encoder and an acoustic encoder, which process the input from the optical camera respectively. Input of sonar equipment ; The photoacoustic fusion module consists of a photoacoustic cross-attention module and an acousto-photonic cross-attention module, which generate fusion results dominated by optical features and dominated by acoustic features, respectively. The photoacoustic dual-path decoder includes an optical decoder and an acoustic decoder, which process the output from the photoacoustic fusion module, respectively. and And obtain optical prediction images respectively. Harmony prediction diagram The prediction fusion module further... and Perform weighted fusion to generate the final prediction map. .

[0006] Preferably, the optical encoder includes four identical convolutional blocks and three max-pooling downsampling operations, wherein the outputs of the four convolutional blocks are respectively... , , and The outputs of the three max-pooling downsampling operations are respectively , and Convolution block one The input is an optical input image. Convolutional block 2 The input is Convolutional block three The input is Convolutional block four The input is Max pooling downsampling The input is Max pooling downsampling The input is Max pooling downsampling three The input is ; The acoustic encoder consists of four identical convolutional blocks and three max-pooling downsampling operations. The outputs of the four convolutional blocks are as follows: , , and The outputs of the three max-pooling downsampling operations are respectively , and Convolution block one The input is an acoustic input image. Convolutional block 2 The input is Convolutional block three The input is Convolutional block four The input is Max pooling downsampling The input is Max pooling downsampling The input is Max pooling downsampling three The input is ; The convolutional block includes two repetitions. Convolution, batch normalized (BN), and ReLU activation functions.

[0007] Preferably, the input to the photoacoustic fusion module is the output of the optical encoder. The output of the acoustic encoder The photoacoustic cross-attention module first, based on Calculate the optical query vector and according to Calculate the acoustic key vector Sum value vector And use this to calculate photoacoustic cross-attention. ; The audio-visual cross-attention module first based on Calculate the acoustic query vector and according to Calculate the optical bond vector Sum value vector And use this to calculate the acoustic-optical cross-attention. .

[0008] Preferably, the optical decoder includes three skip fusion modules, three upsampling modules, and three convolutional blocks; wherein the outputs of the three skip fusion modules are respectively , and The outputs of the three upsampling modules are respectively , and The outputs of the three convolutional blocks are respectively , and ; The skip fusion module receives the outputs from the upsampling module and the corresponding optical encoder convolutional block, and performs a stitching and fusion operation; the inputs of the three skip fusion modules are respectively and , and , and The inputs of the three upsampling modules are respectively , and The inputs to the three convolutional blocks are respectively , and ; The acoustic decoder consists of three skip fusion modules, three upsampling modules, and three convolutional blocks; the outputs of the three skip fusion modules are respectively... , and The outputs of the three upsampling modules are respectively , and The outputs of the three convolutional blocks are respectively , and ; Its skip fusion module receives the outputs from the upsampling module and the corresponding acoustic encoder convolutional block, and performs a splicing and fusion operation; the inputs of the three skip fusion modules are respectively and , and , and The inputs of the three upsampling modules are respectively , and The inputs to the three convolutional blocks are respectively , and .

[0009] Preferably, the prediction fusion module adjusts the weight allocation of the optical prediction map and the acoustic prediction map during the fusion process based on the degree of degradation of the optical image. The degree of degradation of the optical image is calculated as follows: First, input an RGB optical image. Convert to grayscale image ; Then, the normalized Laplacian variance and contrast of the grayscale image are calculated to represent the degree of degradation, and the weights of the optical and acoustic prediction maps are assigned accordingly, as shown below: ; in, Indicates the Laplace variance; Represents the variance of the image; Represents the Laplacian operator for images; Indicates image contrast; and These represent the optical image fusion weights and the acoustic image fusion weights, respectively. and These represent the upper limit of the Laplacian variance and the upper limit of the contrast of the image, respectively. The final predicted image is calculated as follows: .

[0010] Preferred, constructed loss function Including photoacoustic consistency loss and monitoring losses The details are as follows: ; in, Represents the Dice loss function; , , and Let represent the optical prediction map, acoustic prediction map, final prediction map, and ground truth label for the i-th image, respectively. This represents the weighting of the loss function.

[0011] Compared with the prior art, the beneficial effects of this application are as follows: 1. Fully utilize the complementarity of optical and acoustic images. Most existing technologies rely on single-modal data for segmentation, which is easily affected by factors such as imaging quality and noise, thus impacting segmentation performance. In contrast, this application integrates optical and acoustic features through an optical-acoustic fusion module and ensures the consistency of the target region in the multimodal prediction map through photoacoustic consistency loss, fully leveraging the synergistic advantages of both and compensating for the shortcomings of single-modal data.

[0012] 2. Adaptively adjust the segmentation strategy based on the degree of optical image degradation. This application calculates the normalized Laplacian variance and contrast of optical images as estimates of image degradation, and then dynamically adjusts the weight allocation of the final fused prediction image. When the optical image quality is poor, the model automatically increases the weight of acoustic image prediction, ensuring the robustness of prediction results in various deep-sea environments. Attached Figure Description

[0013] Figure 1 A segmentation model for X-type multimodal deep-sea polymetallic nodules; Figure 2 It is a photoacoustic fusion module. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0015] This application proposes a deep-sea polymetallic nodule image segmentation method based on multimodal data fusion. This method effectively improves the segmentation performance of deep-sea nodule regions by integrating features from photoacoustic modal images. The specific technical solution includes steps such as image acquisition, image registration, dataset construction, construction of an X-shaped multimodal deep-sea polymetallic nodule segmentation model, model training, and model application. The detailed technical implementation is as follows: 1. Image Acquisition: During the image acquisition phase, underwater optical cameras and sonar equipment are used to simultaneously acquire optical and acoustic images.

[0016] 2. Image registration: In the image registration stage, optical and acoustic images are registered to obtain photoacoustic image pairs with consistent resolution and spatial and temporal uniformity.

[0017] 3. Dataset Construction: In the data set construction phase, ground truth labels are added to the acquired photoacoustic images to construct the dataset.

[0018] 4. Construction of an X-type multimodal deep-sea polymetallic nodule segmentation model: The X-type multimodal deep-sea polymetallic nodule segmentation model consists of a photoacoustic dual-path encoder, a photoacoustic fusion module, and a photoacoustic dual-path decoder.

[0019] ① The photoacoustic dual-path encoder processes the input from the optical camera through two parallel coding networks. Input of sonar equipment .

[0020] The optical encoder consists of four identical convolutional blocks (labeled as follows) ) and 3 max pooling downsampling ( The outputs of the four convolutional blocks are as follows: , , and The outputs of the three max-pooling downsampling operations are respectively , and Convolutional block one The input is an optical input image. Convolutional block 2 The input is Convolutional block three The input is Convolutional block four The input is Max pooling downsampling The input is Max pooling downsampling The input is Max pooling downsampling three The input is .

[0021] The acoustic encoder consists of four identical convolutional blocks ( , , , ) and 3 max pooling downsampling ( , , The outputs of the four convolutional blocks are as follows: , , and The outputs of the three max-pooling downsampling operations are respectively , and Convolutional block one The input is an acoustic input image. Convolutional block 2 The input is Convolutional block three The input is Convolutional block four The input is Max pooling downsampling The input is Max pooling downsampling The input is Max pooling downsampling three The input is .

[0022] The convolutional block consists of two repetitions. It consists of convolution, batch normalization (BN), and ReLU activation functions.

[0023] ②The optical-acoustic fusion module consists of an optical-acoustic cross-attention module and an acoustic-optical cross-attention module, which respectively generate fusion results dominated by optical features and dominated by acoustic features.

[0024] The input to the photoacoustic fusion module is the output of the optical encoder. The output of the acoustic encoder The photoacoustic cross-attention module first, based on... Calculate the optical query vector and according to Calculate the acoustic key vector Sum value vector And use this to calculate photoacoustic cross-attention. The audio-visual cross-attention module first, based on... Calculate the acoustic query vector and according to Calculate the optical bond vector Sum value vector And use this to calculate the acoustic-optical cross-attention. .

[0025] ③ The optical-acoustic dual-path decoder processes the output from the optical-acoustic fusion module through two parallel decoding networks. and And obtain optical prediction images respectively. Harmony prediction diagram The prediction fusion module further... and Perform weighted fusion to generate the final prediction map. .

[0026] The optical decoder comprises three skip fusion modules, three upsampling modules, and three convolutional blocks. The outputs of the three skip fusion modules are as follows: , and The outputs of the three upsampling modules are respectively , and The outputs of the three convolutional blocks are respectively , and The skip fusion module receives the outputs from the upsampling module and the corresponding optical encoder convolutional block, and then concatenates and fuses them. The inputs to the three skip fusion modules are as follows: and , and , and The inputs to the three upsampling modules are respectively , and The inputs to the three convolutional blocks are respectively , and .

[0027] The acoustic decoder consists of three skip fusion modules, three upsampling modules, and three convolutional blocks. The outputs of the three skip fusion modules are as follows: , and The outputs of the three upsampling modules are respectively , and The outputs of the three convolutional blocks are respectively , and The skip fusion module receives the outputs from the upsampling module and the corresponding acoustic encoder convolutional block, and then concatenates and fuses them. The inputs to the three skip fusion modules are as follows: and , and , and The inputs to the three upsampling modules are respectively , and The inputs to the three convolutional blocks are respectively , and .

[0028] The predictive fusion module adjusts the weight distribution of the optical and acoustic prediction maps during the fusion process based on the degree of degradation of the optical image. When the optical image quality is good, the weight of optical prediction is increased and the influence of acoustic prediction is reduced; when the optical image quality is poor, the weight of acoustic prediction is increased and the weight of optical prediction is reduced. Specifically, the calculation of the degree of optical image degradation first involves inputting the RGB optical image... Convert to grayscale image Then, the normalized Laplacian variance and contrast of the grayscale image are calculated to represent the degree of degradation, and the weights of the optical and acoustic prediction maps are assigned accordingly, as shown below: ; in, Indicates the Laplace variance; Represents the variance of the image; Represents the Laplacian operator for images; Indicates image contrast; and These represent the optical image fusion weights and the acoustic image fusion weights, respectively. and These represent the upper limit of the Laplacian variance and the upper limit of the contrast of the image, respectively.

[0029] The final predicted image is calculated as follows: .

[0030] 5. Model Training: Construct a loss function and train the model built in step 4 using the dataset constructed in step 3, minimizing the loss function. The constructed loss function... Loss of photoacoustic consistency and monitoring losses The components are as follows: ; in, Represents the Dice loss function; , , and Let represent the optical prediction map, acoustic prediction map, final prediction map, and ground truth label for the i-th image, respectively. This represents the weighting of the loss function.

[0031] Model Application: Based on the model trained in step 5, input the deep-sea polymetallic nodule image to be segmented, and the polymetallic nodule region will be automatically segmented.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep-sea polymetallic nodule image segmentation method based on multi-modal data fusion, characterized in that, Includes the following steps: S1. Acquire image; S2. Image registration; S3. Construct a dataset based on the acquired photoacoustic images; S4. Construct an X-shaped multimodal deep-sea polymetallic nodule segmentation model; The X-type multimodal deep-sea polymetallic nodule segmentation model includes a photoacoustic dual-path encoder, a photoacoustic fusion module, and a photoacoustic dual-path decoder. The photoacoustic dual-path encoder includes an optical encoder and an acoustic encoder that process inputs from an optical camera and a sonar device, respectively ;​ The photoacoustic fusion module consists of a photoacoustic cross-attention module and an acousto-photonic cross-attention module, which generate fusion results dominated by optical features and dominated by acoustic features, respectively. The photoacoustic dual-path decoder includes an optical decoder and an acoustic decoder, which respectively process the outputs from the photoacoustic fusion module and and respectively obtain an optical prediction map and an acoustic prediction map The prediction fusion module further performs weighted fusion on and and generates a final prediction map ; S5. Construct a loss function and train the model constructed in step S4 using the dataset constructed in step S3 to minimize the loss function; S6. Using the model trained in step S5, input the deep-sea polymetallic nodule image to be segmented, and automatically segment out the polymetallic nodule region; The predictive fusion module adjusts the weight distribution of the optical and acoustic prediction maps during the fusion process based on the degree of degradation of the optical image. The degree of degradation of the optical image is calculated as follows: First, input an RGB optical image. Convert to grayscale image ; Then, the normalized Laplacian variance and contrast of the grayscale image are calculated to represent the degree of degradation, and the weights of the optical and acoustic prediction maps are assigned accordingly, as shown below: ； in, Indicates the Laplace variance; Represents the variance of the image; Represents the Laplacian operator for images; Indicates image contrast; and These represent the optical image fusion weights and the acoustic image fusion weights, respectively. and These represent the upper limit of the Laplacian variance and the upper limit of the contrast of the image, respectively. The final predicted image is calculated as follows: 。 2. The deep-sea polymetallic nodule image segmentation method based on multimodal data fusion according to claim 1, characterized in that, In step S2, during the image registration stage, the optical image and the acoustic image are registered to obtain photoacoustic image pairs with consistent resolution and spatial-temporal uniformity.

3. The deep-sea polymetallic nodule image segmentation method based on multimodal data fusion according to claim 1, characterized in that, The optical encoder consists of four identical convolutional blocks and three max-pooling downsampling operations, where the outputs of the four convolutional blocks are respectively... , , and The outputs of the three max-pooling downsampling operations are respectively , and Convolution block one The input is an optical input image. Convolutional block 2 The input is Convolutional block three The input is Convolutional block four The input is Max pooling downsampling The input is The input of max pooling downsampling 2 is Max pooling downsampling three The input is ; The acoustic encoder consists of four identical convolutional blocks and three max-pooling downsampling operations. The outputs of the four convolutional blocks are as follows: , , and The outputs of the three max-pooling downsampling operations are respectively , and Convolution block one The input is an acoustic input image. Convolutional block 2 The input is Convolutional block three The input is Convolutional block four The input is Max pooling downsampling The input is Max pooling downsampling The input is Max pooling downsampling three The input is ; The convolutional block includes two repetitions. Convolution, batch normalized (BN), and ReLU activation functions.

4. The deep-sea polymetallic nodule image segmentation method based on multimodal data fusion according to claim 3, characterized in that, The input to the photoacoustic fusion module is the output of the optical encoder. The output of the acoustic encoder The photoacoustic cross-attention module first, based on Calculate the optical query vector and according to Calculate the acoustic key vector Sum value vector And use this to calculate photoacoustic cross-attention. ; The audio-visual cross-attention module first based on Calculate the acoustic query vector and according to Calculate the optical bond vector Sum value vector And use this to calculate the acoustic-optical cross-attention. .

5. The deep-sea polymetallic nodule image segmentation method based on multimodal data fusion according to claim 3, characterized in that, The optical decoder includes three skip fusion modules, three upsampling modules, and three convolutional blocks; the outputs of the three skip fusion modules are respectively... , and The outputs of the three upsampling modules are respectively , and The outputs of the three convolutional blocks are respectively , and ; The skip fusion module receives the outputs from the upsampling module and the corresponding optical encoder convolutional block, and performs a stitching and fusion operation; the inputs of the three skip fusion modules are respectively and , and , and The inputs of the three upsampling modules are respectively , and The inputs to the three convolutional blocks are respectively , and ; The acoustic decoder consists of three skip fusion modules, three upsampling modules, and three convolutional blocks; the outputs of the three skip fusion modules are respectively... , and The outputs of the three upsampling modules are respectively , and The outputs of the three convolutional blocks are respectively , and ; Its skip fusion module receives the outputs from the upsampling module and the corresponding acoustic encoder convolutional block, and performs a splicing and fusion operation; the inputs of the three skip fusion modules are respectively and , and , and The inputs of the three upsampling modules are respectively , and The inputs to the three convolutional blocks are respectively , and .

6. The deep-sea polymetallic nodule image segmentation method based on multimodal data fusion according to claim 1, characterized in that, Constructed loss function Including photoacoustic consistency loss and monitoring losses The details are as follows: ； in, Represents the Dice loss function; , , and Let represent the optical prediction map, acoustic prediction map, final prediction map, and ground truth label for the i-th image, respectively. This represents the weighting of the loss function.

Images