Method for identifying seawater aging microplastics based on multi-modal deep learning
By fusing Raman spectroscopy and microscopic image features through multimodal deep learning, the accuracy and robustness issues of single-modal models in the identification of aged microplastics were solved, achieving high-precision and stable identification of the degree and type of microplastic aging.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES)
- Filing Date
- 2026-04-10
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies, when identifying microplastics in aged seawater, suffer from decreased accuracy and poor robustness due to the use of single-modal models, making it difficult to comprehensively reflect the aging state.
A multimodal deep learning approach is adopted, combining visual and spectral branches. A dual-branch deep learning model with lightweight improved convolutional neural network and sparse principal component analysis for dimensionality reduction is used to fuse Raman spectral and microscopic image features. Feature fusion is performed using adaptive weighting and attention mechanisms, and finally recognition is achieved through a Softmax classifier.
It achieves high-precision identification of aged microplastics, with an accuracy rate of over 96%. It maintains robustness and can still identify stably when the spectral signal-to-noise ratio is reduced or the image is occluded. It also has highly automated end-to-end analysis capabilities.
Smart Images
Figure CN122347804A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of environmental monitoring and artificial intelligence technology, and in particular to a method for identifying microplastics in seawater aging based on multimodal deep learning. Background Technology
[0002] Current methods for detecting microplastics in seawater largely rely on single-spectral techniques such as Raman and infrared spectroscopy. However, in the marine environment, microplastics undergo aging processes including photo-oxidation, hydrolysis, and bioattachment, resulting in complex changes in both their surface morphology and chemical structure. This leads to a significant decrease in the accuracy of identification methods based on standard spectral libraries. Relying solely on Raman spectroscopy cannot effectively capture the morphological changes during the aging process of microplastics, and relying solely on visual image analysis makes it difficult to accurately perceive the evolution of their internal chemical structure.
[0003] Current models built on a single modality can usually achieve good recognition results under ideal conditions or on standard microplastic samples. However, when faced with aged microplastic samples, the recognition performance of the model will decline, making it difficult to fully reflect the aging state and resulting in poor robustness. Summary of the Invention
[0004] To overcome the aforementioned problems in the existing technology, this invention proposes a method for identifying microplastics in seawater aging based on multimodal deep learning.
[0005] The technical solution adopted by this invention to solve its technical problem is: a method for identifying seawater aging microplastics based on multimodal deep learning, comprising the following steps: Step 1: Prepare various seawater microplastic samples and conduct aging experiments on the microplastic samples. During the aging experiment, periodically collect spectral data and microscopic images of the samples to construct a dataset. Step 2: Denoise reduction, baseline correction, normalization, and dimensionality reduction are performed on the collected spectral data, and image processing algorithms are used to process the microscopic images. Step 3: Construct a dual-branch deep learning model. The visual branch adopts a lightweight improved convolutional neural network architecture, while the spectral branch adopts an architecture that combines sparse principal component analysis dimensionality reduction with a fully connected neural network. The feature fusion module fuses the features of the two branches and outputs the recognition result through a classifier. Step 4: Divide the dataset from Step 1 into a training set, a validation set, and a test set, and train the model obtained in Step 3 to obtain the final recognition model.
[0006] The above-mentioned method for identifying seawater aging microplastics based on multimodal deep learning, specifically step 1, involves: preparing various seawater microplastic samples to be identified, laying them flat and exposing them to an ultraviolet experimental device, and cyclically performing drying, seawater spraying, and condensation. Spectral data and microscopic images of the samples are collected periodically, and microscopic images of the microplastics are simultaneously collected at the same location where spectral data is collected each time.
[0007] The above-mentioned method for identifying microplastics in seawater aging based on multimodal deep learning, specifically step 2, involves: for spectral data, denoising the acquired spectral data using wavelet transform; performing baseline correction on the spectral data using iterative adaptive weighted penalized least squares method; eliminating spectral intensity differences through normalization; and performing dimensionality reduction on the preprocessed spectral data using sparse principal component analysis. For microscopic images, image processing algorithms are used to process the images, specifically including grayscale value, median filtering for noise reduction, and adaptive threshold segmentation to extract microplastic contours.
[0008] The above-mentioned method for identifying seawater aging microplastics based on multimodal deep learning, in step 3, the visual branch of the dual-branch deep learning model adopts a CNN neural network model to extract the morphological aging features of microplastics and output a visual feature vector with a fixed dimension; the spectral branch adopts an FCN neural network model, inputting the spectral data reduced by sparse principal component analysis into the FCN neural network model and outputting a spectral feature vector with the same dimension as the visual branch.
[0009] The aforementioned method for identifying seawater aging microplastics based on multimodal deep learning employs an adaptive weighted fusion strategy in its feature fusion module. Combined with a channel attention mechanism, the fusion weights of the two types of features are dynamically allocated based on the difference in representational capabilities between visual and spectral features, thereby achieving complementary fusion of the two types of aging features. The attention mechanism enhances the focus on key aging features.
[0010] The above-mentioned method for identifying seawater aging microplastics based on multimodal deep learning uses a Softmax classifier as the classifier and inputs a fused feature vector optimized by an attention mechanism to complete the dual task of identifying the degree of microplastic aging and the type of microplastic.
[0011] The beneficial effects of the present invention are: (1) High recognition accuracy: By integrating morphological information and spectral information, the recognition accuracy of aged microplastics is increased to more than 96%; (2) Strong robustness: The dual-modal information complements each other. When the spectral signal-to-noise ratio decreases or the image is partially occluded, the model can still maintain stable recognition. (3) High degree of automation: It can realize end-to-end automated analysis from data input to result output. Attached Figure Description
[0012] Figure 1 This is a schematic diagram of the process of this invention; Figure 2 This is a diagram of the dual-branch deep learning model fusion architecture of the present invention. Detailed Implementation
[0013] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0014] This invention discloses a method for identifying microplastics in seawater aging based on multimodal deep learning. It achieves microplastic detection and identification through five steps: "sample collection → spectral and image data acquisition → data preprocessing → model training and construction → model verification and optimization → classification performance testing". The core is the multimodal deep fusion of multidimensional Raman spectral features and visual image features.
[0015] Specific steps are as follows Figure 1 As shown, it includes: (a) Sample collection 1. Sample preparation: Ten common seawater microplastics were selected as research subjects, including: polyamide, polybutylene terephthalate, polycarbonate, polyethylene, polypropylene, polystyrene, polytetrafluoroethylene, polyvinyl chloride, polyurethane, and acrylonitrile. All samples were prepared into uniformly sized particles, spread out, and exposed to ultraviolet light at a wavelength of 340 nm and a spectral irradiance of 1200 W / m². 2 The process involved 8 hours of drying, 0.25 hours of spraying (artificial seawater), and 3.75 hours of condensation, with a total aging time of 30 days. Samples were taken every 2 days, with a total of 150 samples collected for each sample.
[0016] 2. Data Collection: At each sampling time point, the aged sample was placed on the sample stage of the Raman spectrometer, and the detection parameters were set as follows: excitation wavelength 785 nm, laser power 400 mW, and spectral range 200-2500 cm⁻¹. -1 The integration time was 3 seconds. Spectral data for each sample was randomly collected at different locations under 4x, 10x, and 20x magnification. At the same location where Raman spectra were collected each time, surface morphology images of the microplastics were simultaneously acquired to ensure a one-to-one correspondence between the images and the spectral data.
[0017] (ii) Data preprocessing 1. Spectral processing: The acquired spectral data is denoised using a wavelet transform method (db4 wavelet, 5 decomposition levels); baseline correction is performed on the spectral data using an iterative adaptive weighted penalized least squares method (airPLS); spectral intensity differences are eliminated using Min-Max normalization; and dimensionality reduction is performed on the preprocessed spectral data using sparse principal component analysis (SPCA), which reduces the dimensionality to 128 dimensions in this embodiment to reduce the computational load of the model.
[0018] 2. Image Processing: The image was converted to grayscale; noise was removed using median filtering (3×3 window); microplastic contours were extracted using adaptive thresholding; morphological parameters (such as area, perimeter, and roundness) and texture features (such as contrast, entropy, and energy) were extracted to quantify the morphological evolution during the aging process of microplastics.
[0019] (III) Construction of dual-branch deep learning models (e.g.) Figure 2 (As shown) 1. Dual-branch network architecture design: The visual branch adopts a lightweight improved convolutional neural network (CNN) architecture. It takes a preprocessed microplastic morphology image as input and automatically extracts morphological aging features such as surface cracks and roughness of microplastics through the synergistic effect of convolution, pooling and activation layers. Finally, it outputs a fixed-dimensional visual feature vector.
[0020] In this embodiment, the visual branch uses an improved ResNet-18, with an input image size of 224x224 and an output 128-dimensional feature vector.
[0021] The spectral branch adopts a "SPCA dimensionality reduction + FCN" combination structure. The spectral data reduced by SPCA is input into a fully connected neural network (FCN) to further mine deep spectral aging features such as feature peak shift and peak intensity change, and outputs a spectral feature vector with the same dimension as the visual branch to facilitate fusion operation.
[0022] In this embodiment, the spectral branch uses a three-layer fully connected network (128-256-128), with 128-dimensional reduced spectral data as input and 128-dimensional feature vectors as output.
[0023] 2. Feature fusion module design: An adaptive weighted fusion strategy is adopted, combined with a channel attention mechanism to construct a fusion module. Based on the difference in representational capabilities between visual and spectral features, the fusion weights of the two types of features are dynamically allocated to achieve complementary fusion of the two types of aging features. The attention mechanism enhances the model's focus on key aging features, effectively suppressing redundant information such as background noise and irrelevant interference, and improving the relevance and effectiveness of feature representation.
[0024] 3. Classifier and overall model architecture: A Softmax classifier is selected, and the fused feature vector optimized by the attention mechanism is input to complete the dual task of identifying the aging degree and type of microplastics. The visual branch, spectral branch, feature fusion module and classifier are sequentially connected and integrated to form a complete multimodal recognition model for aging microplastics.
[0025] (iv) Model training and optimization 1. Dataset partitioning: The preprocessed spectral dataset and image dataset are matched one-to-one, and the datasets are divided into training set, validation set and test set in a ratio of 7:2:1.
[0026] 2. Model Training: The Adam optimizer is used, along with the cross-entropy loss function. Core parameters such as the number of training epochs are set (in this embodiment, the learning rate is 0.0001; the loss function is cross-entropy loss; the batch size is 32; and the number of training epochs is 200). An early stopping mechanism is introduced (training stops when the validation set loss does not decrease for 20 consecutive epochs). The dropout rate is 0.5, and the weight decay is 1×10⁻⁶. -4 To suppress overfitting, the preprocessed dataset is input into the model during training. Features are extracted through a two-branch process, and the prediction results are output after fusion and optimization. The parameters of each module are iteratively adjusted using the backpropagation algorithm, and the model convergence is monitored using the validation set to ensure that the model with the best performance on the validation set is selected.
[0027] 3. Model Optimization: Focusing on improving recognition accuracy and generalization ability, we compared the fusion effects of different attention mechanisms, optimized the adaptive weight calculation logic, and adjusted the dual-branch network structure and hyperparameters. To address the overfitting problem that may occur during training, we strengthened data augmentation and optimized the regularization strategy. Through multiple rounds of comparative experiments, we determined the optimal combination parameters.
[0028] (v) Model performance verification Performance validation employs accuracy, precision, recall, F1 score, and robustness metrics to comprehensively evaluate model performance. It also compares and validates the model against single-vision and single-spectral-branch models. During validation, the optimal trained model is loaded, test set data is input into the model to obtain prediction results, and various core metrics are calculated to clarify the advantages of the fusion model compared to single-branch models.
[0029] On the test set, the method of this embodiment achieved the performance metrics shown in Table 1.
[0030] Table 1 In robustness tests, when the Raman spectral signal-to-noise ratio decreased to 50% of the original signal, the accuracy of the single spectral model dropped to 71.2%, while the fusion model of the present invention still maintained a recognition accuracy of 92.3%; when the image had 30% occlusion, the accuracy of the single visual model dropped to 68.7%, while the model of the present invention still reached 89.4%.
[0031] This embodiment verifies the effectiveness of the proposed multimodal fusion method in identifying aging microplastics in seawater. By fusing morphological images and Raman spectral information, the model significantly outperforms the single-modal model in terms of recognition accuracy and robustness, enabling simultaneous identification of microplastic types and aging degrees, thus meeting the practical application needs of marine environmental monitoring.
[0032] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.
Claims
1. A method for identifying microplastics in seawater aging based on multimodal deep learning, characterized in that, The steps include the following: Step 1: Prepare various seawater microplastic samples and conduct aging experiments on the microplastic samples. During the aging experiment, periodically collect spectral data and microscopic images of the samples to construct a dataset. Step 2: Denoise reduction, baseline correction, normalization, and dimensionality reduction are performed on the collected spectral data, and image processing algorithms are used to process the microscopic images. Step 3: Construct a dual-branch deep learning model. The visual branch adopts a lightweight improved convolutional neural network architecture, while the spectral branch adopts an architecture that combines sparse principal component analysis dimensionality reduction with a fully connected neural network. The feature fusion module fuses the features of the two branches and outputs the recognition result through a classifier. Step 4: Divide the dataset from Step 1 into a training set, a validation set, and a test set, and train the model obtained in Step 3 to obtain the final recognition model.
2. The method for identifying seawater aging microplastics based on multimodal deep learning according to claim 1, characterized in that, Step 1 specifically involves: preparing various seawater microplastic samples to be identified, laying them flat and exposing them to an ultraviolet experimental device, and cyclically performing drying, seawater spraying, and condensation. Spectral data and microscopic images of the samples are collected periodically, and microscopic images of the microplastics are simultaneously collected at the same location where spectral data is collected each time.
3. The method for identifying seawater aging microplastics based on multimodal deep learning according to claim 1, characterized in that, Step 2 specifically involves: for spectral data, performing noise reduction processing on the acquired spectral data based on wavelet transform; and performing baseline correction processing on the spectral data based on iterative adaptive weighted penalized least squares method. Spectral intensity differences are eliminated by normalization. Sparse principal component analysis was used to reduce the dimensionality of the preprocessed spectral data. For microscopic images, image processing algorithms are used to process the images, specifically including grayscale value, median filtering for noise reduction, and adaptive threshold segmentation to extract microplastic contours.
4. The method for identifying seawater aging microplastics based on multimodal deep learning according to claim 1, characterized in that, In step 3, the visual branch of the dual-branch deep learning model uses a CNN neural network model to extract the morphological aging features of microplastics and output a visual feature vector with a fixed dimension; the spectral branch uses an FCN neural network model to input the spectral data reduced by sparse principal component analysis into the FCN neural network model and output a spectral feature vector with the same dimension as the visual branch.
5. The method for identifying seawater aging microplastics based on multimodal deep learning according to claim 4, characterized in that, The feature fusion module adopts an adaptive weighted fusion strategy, combined with a channel attention mechanism. Based on the difference in representational capabilities between visual and spectral features, it dynamically allocates the fusion weights of the two types of features to achieve complementary fusion of the two types of aging features. The attention mechanism enhances the focus on key aging features.
6. The method for identifying seawater aging microplastics based on multimodal deep learning according to claim 5, characterized in that, The classifier used is the Softmax classifier. The input is a fused feature vector optimized by the attention mechanism, which completes the dual task of identifying the degree of microplastic aging and the type of microplastic.