Cross-modal sample matching method and system applied to micro-plastic image recognition

Abstract

The invention provides a cross-modal sample matching method applied to micro-plastic image recognition. The cross-modal sample matching method comprises the steps that particle position extraction and data pair construction are carried out on a micro-plastic image; constructing a double-input and backbone-sharing neural network model which is set to receive particle sample images of a reflected light mode and a transmission light mode, and outputting corresponding particle identifiers and high-dimensional feature vectors; training the neural network model by using the particle sample images of the two modalities forming the data pair to obtain a matching model; and in the reasoning stage, cross-modal matching is performed on the to-be-queried image and the library image by utilizing the matching model, and a final matching result is obtained. According to the method, matching of high-dimensional feature vectors of two groups of images in different modalities is realized through a neural network model with double-flow input and a shared backbone, and matching of shot micro-plastic images in different modalities is realized, so that accurate substance types and enough clear images are provided for an image recognition model for training at the same time.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine vision, and particularly relates to a cross-modal sample matching method and system applied to micro-plastic image recognition. BACKGROUND

[0002] Mask: In image processing, a mask is a binary image composed of 0 and 1. When a mask is applied in a certain function, the 1 value area is processed, while the 0 value area that is shielded is not included in the calculation. A mask is a special image whose pixel value has only two states of 0 and 1, and is often used to realize the logical operation of an image.

[0003] Current models applied to micro-plastic related research are divided into two types: machine learning models (ML) and deep learning models (DL). Machine learning classification algorithms have been widely used in MP (micro-plastic) analysis, such as decision tree (DT), random forest (RF), support vector machine, k-nearest neighbor (KNN), etc. These machine learning classification algorithms aim to extract feature information between MP data samples and classify and identify MPs based on the feature information.

[0004] Deep learning is a subfield of machine learning models, commonly using artificial neural networks (ANN) and convolutional neural networks (CNN), which are suitable for classification or regression tasks of numerical data. It can identify micro-plastic features by capturing complex nonlinear relationships, and the convolutional neural network has derived many models suitable for micro-plastic image recognition, such as Fast R-CNN, U Net, Mask CNN, and YOLO, etc.

[0005] These models have different purposes, and their data set acquisition methods also differ. The mainstream approach is to use a camera to take micro-plastic pictures and use a spectrometer to take micro-plastic spectra as a data set, which also leads to different problems.

[0006] For models using micro-plastic image data sets, taking clear micro-plastic particle images alone cannot obtain accurate material information through spectroscopy, which makes such models still remain in morphological classification, which is summarized as particles, fibers, films, etc., and lacks a unified and clear judgment standard, which is insufficient to further determine the specific type of a certain micro-plastic particle. Hartmann et al. defined films as planar particles, whose one-dimensional size is significantly smaller than the other two dimensions, but this shape is only applicable to particles with a particle size > 300 μm; Liu et al. used focus group discussion method, which completely used expert opinions as samples to comprehensively judge the morphology of micro-plastics, and did not involve size judgment basis.

[0007] For models with spectra as data sets, although the substance information can be accurately obtained, no judgment is made in combination with pictures of microplastics, which also leads to problems such as inability to intuitively judge whether it is microplastic and its category.

[0008] The image recognition model is one of deep learning models. The existing image recognition model needs to provide images and labels of microplastics for training. The microplastic labels and high-definition pictures are obtained by a spectrum instrument and a microscope respectively. However, at present, there is no method for matching the images obtained by the spectrum instrument and the microscope, which leads to difficulty in simultaneously providing accurate substance categories (spectrum images converted into text labels) and sufficiently clear images to the image recognition model for training, so that accurate recognition of microplastics cannot be achieved, or a large amount of manpower is needed for artificial classification, leading to difficulty in application of the model.

[0009] The current matching algorithm mainly includes the following two matching algorithms:

[0010] Contour matching algorithm is a classic computer vision technique, and its core is to analyze and compare the geometric characteristics of the outer boundary of an object to judge the similarity of the shape. The standard process of this method first needs to pre-process and segment the original image, such as separating the target particles from the background by thresholding or edge detection algorithm, so as to obtain a clear object boundary. Then, the algorithm tracks these boundary pixel points to form an ordered point set to accurately describe the contour of the particle. In order to facilitate quantitative comparison, this contour point set is converted into a compact mathematical form, i.e. "shape descriptor", and common methods include Hu moment or Fourier descriptor which are not sensitive to translation, scaling and rotation. Finally, by calculating the distance (such as Euclidean distance) between the contour descriptors of two different objects, the system can get a similarity score, and the higher the score, the more matched the shapes of the two objects.

[0011] Feature point matching algorithm:

[0012] The technical core of the feature point matching algorithm is to find stable local key points (such as corner points and spots) in the image which are not sensitive to scale, rotation and brightness changes. The algorithm first detects these key points in two images to be matched, then generates a representative feature descriptor for the local area around each key point. Finally, by comparing the similarity of the feature descriptors in the two images, a one-to-one correspondence between the key points is found and established, so as to realize the matching of the images.

[0013] The disadvantage of the existing matching algorithm is that the existing matching algorithm is only for matching the same type of image, and there is no matching technology directly adapting the shooting results of two instruments. The shooting results of the spectral instrument and the microscope in some matching algorithms have various differences, including but not limited to picture definition difference, particle position change and background difference caused by different light sources, etc., and cannot be directly matched by contour matching algorithm and feature point matching algorithm.

[0014] In addition, in other fields of machine vision, such as person re-identification (ReID), there is also a technology for matching the same identity under different cameras, and the core problem is to solve the identity association of pedestrians under different camera imaging conditions. However, the design goal of the ReID technology is to target macroscopic pedestrian targets (the main difference is posture, occlusion, background), and the image preprocessing (such as cropping based on the human detection box), feature extraction (focusing on the clothing texture and human contour backbone network) are essentially different from the microplastic particle matching problem to be solved by the present application: microplastic particles are microscopic small targets, and the imaging difference is caused by the boundary blur and background noise interference caused by the light source type, rather than the posture or angle, so the existing ReID technology cannot be directly migrated and applied.

[0015] In summary, there is no traditional matching technology directly adapting the shooting results of two instruments, and there is no existing technology applying the related deep learning idea to solve the cross-modal sample matching problem caused by different imaging principles in the field of microplastics. SUMMARY

[0016] The purpose of the present application is to provide a cross-modal sample matching method and system applied to microplastic image recognition, to solve the matching between the shooting results of different shooting methods, improve the application of old technology and create a new technology system, and solve the problem that the current measurement of microplastics is not systematic.

[0017] In order to achieve the above purpose, the present application provides a cross-modal sample matching method applied to microplastic image recognition, which comprises:

[0018] S1: in the training stage, the microplastic image is processed to obtain a particle sample image; and data pairs are constructed according to the particle sample images of two modalities, to obtain the particle sample images of two modalities constituting the data pairs;

[0019] S2: a neural network model with double-flow input and shared backbone is constructed, the neural network model is set to receive the particle sample images of the reflection light modality and the transmission light modality, and output the corresponding particle identification and high-dimensional feature vector;

[0020] S3: training the neural network model by using the particle sample images of the two modalities constituting the data pair, to obtain a matching model;

[0021] S4: in the inference stage, using the matching model to perform cross-modal matching on the query image and the library image to obtain the final matching result.

[0022] The step S1 specifically comprises:

[0023] S11: extracting the particle position of the microplastic image containing a plurality of microplastic particles, and the microplastic image of a single microplastic particle directly as a particle sample image;

[0024] S12: constructing and enhancing the data pair of the particle sample images of the reflection light modality and the transmission light modality, to obtain the particle sample images of the reflection light modality and the transmission light modality as the data pair;

[0025] The step S11 comprises:

[0026] S101: converting the input microplastic image into a single-channel grayscale image;

[0027] S102: performing filtering processing on the single-channel grayscale image to suppress noise and smooth the image;

[0028] S103: performing image segmentation on the filtered microplastic image to obtain a preliminary binary mask;

[0029] S104: performing a contour finding algorithm on the preliminary binary mask to identify the contours of all independent and closed foreground regions;

[0030] S105: performing area screening on the identified foreground region contours, and according to a preset area threshold, removing the tiny contours with too small area, and only retaining the remaining contours as effective contours of real particles;

[0031] S106: generating a final binary mask according to the effective contours of the real particles and extracting the particle sample image therefrom.

[0032] In the step S103, the image segmentation adopts one of a dynamic threshold segmentation method, an adaptive threshold segmentation, a region growing algorithm, and an independent semantic segmentation network; and the step S106 specifically includes: filling an effective contour of the real particle and drawing to a blank canvas to form a final binary mask; calculating a minimum circumscribed rectangle of the effective contour of the real particle to obtain a bounding box coordinate; and using the final binary mask and the bounding box coordinate to extract two types of training samples from the original colorful microplastic image as particle sample images: ① performing a bitwise AND operation using the final binary mask to extract a particle region of interest image completely shielded by the background, so as to extract information corresponding to the effective contour position; and ② using the bounding box coordinate to cut a rectangular region corresponding to the bounding box to obtain a bounding box image containing the particle and part of the adjacent background.

[0033] In the step S2, the neural network model comprises: a double-flow input module, a shared backbone network, a feature processing module and an output module connected in sequence, the double-flow input module comprises two initial feature extraction modules independent of each other; the shared backbone network comprises a remaining feature extraction module sharing parameters; the feature processing module comprises an adaptive average pooling layer and a batch normalization layer; and the output module comprises a particle classification output branch and a high-dimensional feature output branch for a training stage, and a high-dimensional feature output branch for a reasoning stage.

[0034] The neural network model comprises an initial feature extraction module and a remaining feature extraction module; the initial feature extraction module and the remaining feature extraction module are one of the initial feature extraction modules of an EfficientNet network model, a residual network series, a visual attention transformer, a densely connected network series and a lightweight network series.

[0035] The particle classification output branch for the training stage is configured to send the high-dimensional feature vector into a linear classifier to output a prediction result of particle classification, which is used to calculate a sample discrimination loss; the high-dimensional feature output branch for the training stage is configured to directly retain the high-dimensional feature vector from the feature processing module as a high-dimensional feature vector, which is used to calculate a cross-modal alignment measurement loss; and the high-dimensional feature output branch for the reasoning stage is configured to send the high-dimensional feature vector from the feature processing module into an L2 norm normalization layer for processing to generate a final high-dimensional feature matching vector for matching; the sample discrimination loss is calculated using a standard cross-entropy loss function; and the cross-modal alignment measurement loss comprises at least one of a contrast loss, a triplet loss, an N-pair loss, a multi-similarity loss, a circle loss, a center loss and a ring loss.

[0036] In the step S3, the sample discrimination loss and the cross-modal alignment measurement loss are taken as a composite objective function, and the parameters of the neural network model are optimized through the composite objective function to realize effective alignment of the cross-modal features.

[0037] The step S3 specifically includes:

[0038] S31: performing structured data sampling on the particle sample images of the two modalities constituting the data pair to obtain structured batch data;

[0039] S32: inputting the structured batch data into the neural network model to perform forward propagation and synchronously outputting the prediction result of the particle classification and the high-dimensional feature vector;

[0040] S33: determining the sample discrimination loss and the cross-modal alignment measurement loss according to the prediction result of the particle classification and the high-dimensional feature vector to serve as a composite objective function;

[0041] S34: linearly combining the sample discrimination loss and the cross-modal alignment measurement loss according to a preset weight coefficient to form a total objective function of a single iteration ;

[0042] S35: adjusting the parameters of the neural network model based on the total objective function;

[0043] S36: repeating the steps S31 to S35 to perform iterative training until a convergence judgment condition is met, and a matching model is obtained at this time.

[0044] The step S4 specifically includes:

[0045] S41: processing the to-be-queried image and the library image to obtain particle sample images of the two modalities, and performing feature extraction on the particle sample images by using the matching model to obtain matching high-dimensional feature vectors of the to-be-queried image and the library image as a query feature vector and a library feature matrix;

[0046] S42: performing similarity calculation and threshold screening according to the query feature vector and the library feature matrix to obtain effective candidate matching items of the to-be-queried image;

[0047] S43: determining a final matching result according to the effective candidate matching items.

[0048] On the other hand, the application provides a cross-modal matching system applied to micro-plastic image recognition, comprising:

[0049] a particle extraction module configured to: in a training phase and an inference phase, perform particle position extraction on a microplastic image containing a plurality of microplastic particles to obtain a particle sample image, and in the training phase, construct a data pair according to the particle sample images of the two modalities to obtain the particle sample images of the two modalities constituting the data pair;

[0050] a model training module configured to: train a neural network model by using the particle sample images of the two modalities constituting the data pair to obtain a matching model; the neural network model is a dual-flow input, shared backbone neural network model configured to receive the particle sample images of the reflection light modality and the transmission light modality and output corresponding particle identification and a high-dimensional feature vector;

[0051] a feature extraction and semi-automatic matching module configured to: in the inference phase, perform cross-modality matching on a to-be-queried image and a library image by using the matching model to obtain a final matching result.

[0052] The method of the present application realizes the matching of the high-dimensional feature vectors of the two groups of images of different modalities by using the dual-flow input, shared backbone neural network model, realizes the matching of the microplastic images photographed in different modalities, and thus can provide the image recognition model with accurate material categories (spectrum image converted into text labels) and clear enough images for training; after the two groups of data sets photographed under the spectrometer and the microscope are matched by the cross-modality matching method of the present application and used as the input parameters of the image recognition model, the accurate identification of microplastics can be further realized. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is the flow chart of the particle position extraction and data pair construction of the cross-modality sample matching method applied to the microplastic image recognition of the present application.

[0054] Figure 2 is the structural block diagram of the neural network model constructed by the cross-modality sample matching method applied to the microplastic image recognition of the present application.

[0055] Figure 3 is the flow chart of the model training of the cross-modality sample matching method applied to the microplastic image recognition of the present application.

[0056] Figure 4 is the flow chart of the inference and semi-automatic matching of the cross-modality sample matching method applied to the microplastic image recognition of the present application.

[0057] Figure 5 is the structural block diagram of the cross-modality sample matching system applied to the microplastic image recognition of the present application. DETAILED DESCRIPTION

[0058] The application discloses a cross-modal sample matching method applied to microplastic image recognition, which is used for realizing high-precision semi-automatic matching between shooting results of micro particles to be measured in different modes (i.e. different shooting modes).

[0059] In the embodiment, the different modes applied include a reflection light mode under an optical microscope (as mode A) and a transmission light mode under an infrared spectrometer (as mode B), and in other embodiments, the optical microscope can be replaced by other types of optical imaging instruments such as a super-depth microscope, and the infrared spectrometer can be replaced by other instruments with both spectral and optical imaging functions, for example, a hyperspectral imaging instrument. It should be noted that the instrument replaced by the infrared spectrometer must have both spectral and optical imaging functions, and the reason is that the purpose of matching the image of mode A with higher resolution and the image of mode B with lower resolution to each other in the application is to transfer the spectral information contained in the image of mode B to the image of mode A with higher resolution. The instrument with both spectral and optical imaging functions provides spectral images and optical images for "transferring", and the matching of the "transferred" optical images and the optical images with high-definition particle shapes is realized through the method of the application, and finally used for realizing the matching of particle shapes and material types in the image recognition model.

[0060] The cross-modal sample matching method applied to microplastic image recognition of the application comprises the following steps: step S1, extraction of particle positions and construction of data pairs, step S2, model construction, step S3, model training, and step S4, reasoning and semi-automatic matching.

[0061] Step S1: in the training stage, microplastic image processing is performed to obtain particle sample images; and data pairs are constructed according to the particle sample images of the two modes, to obtain the particle sample images of the two modes constituting the data pairs;

[0062] The purpose of the application is to provide standardized input data with clear features for subsequent deep neural networks from original and complex microscopic images.

[0063] The specific steps of step S1 are as follows:

[0064] Step S11: the particle positions of the microplastic images containing multiple microplastic particles are extracted, and the microplastic images of single microplastic particles are directly used as particle sample images;

[0065] In the present embodiment, the transmission light image is taken separately for each particle due to the limited field of view of the infrared spectrometer, so this part does not need additional processing, only the particle position reading of the reflection light modal image is needed. In other embodiments, if the infrared spectrometer is replaced by an instrument with a larger field of view, the particle position extraction is needed in the case of capturing multiple particles at a time.

[0066] Step S11 is specifically for the reflection light image with complex background to accurately extract the position of the particles.

[0067] Referring to Figure 1 , the step S11 comprises:

[0068] Step S101: converting the input microplastic image (i.e. RGB color image) into a single-channel grayscale image;

[0069] Step S102: filtering the microplastic image converted into a grayscale image to suppress noise and smooth the image;

[0070] Specifically, first, median filtering is performed to suppress salt and pepper noise, and then Gaussian filtering is performed to smooth the image. In this way, the pixel intensity is weakened, laying a foundation for threshold segmentation.

[0071] Step S103: image segmentation is performed on the filtered microplastic image;

[0072] In the present embodiment, the image segmentation adopts a dynamic threshold segmentation method. The dynamic threshold segmentation specifically comprises: based on the grayscale statistical characteristics of all pixels of the microplastic image, a global dynamic threshold is calculated (for example, the global dynamic threshold can be set to 1.6 times the average grayscale value, and in actual application, the multiplier factor of 1.6 times should be adjusted according to the specific circumstances of different images); below the global dynamic threshold is marked as foreground (particle), and above the global dynamic threshold is marked as background, generating a preliminary binary mask.

[0073] Step S104: performing a contour finding algorithm on the preliminary binary mask to identify the contours of all independent and closed foreground regions;

[0074] Step S105: traversing all identified contours and calculating the pixel area enclosed by each contour; according to a pre-set area threshold, the small contours with an area less than the area threshold are regarded as artifacts caused by background impurities or image noise and are removed, and only the contours with an area greater than or equal to the area threshold are retained as effective contours of real particles;

[0075] Step S106: generating a final binary mask according to the effective contours of the real particles and extracting the particle sample image therefrom.

[0076] Step S106 specifically comprises: filling the effective contour of the real particle and drawing to the blank canvas to form the final binary mask; calculating the minimum circumscribed rectangle of the effective contour of the real particle to obtain the Bounding Box (BBOX) coordinates; using the final binary mask and the BBOX coordinates to extract two types of training samples from the original color image as particle sample images: ① using the final binary mask to perform a bitwise AND operation to extract the particle region of interest (Region of Interest, ROI) image completely shielded by the background, thereby extracting the information corresponding to the effective contour position; ② using the BBOX coordinates to crop the rectangular region corresponding to the BBOX to obtain the BBOX image containing the particle and part of the adjacent background. Thus, the obtained particle sample images include the particle region of interest image and the BBOX image. During actual training, one or a mixture of the particle region of interest image and the BBOX image can be selected for use as needed.

[0077] Step S12: data pair construction and enhancement are performed on the particle sample images in the reflection light mode and the transmission light mode to obtain the particle sample images in the reflection light mode and the transmission light mode as data pairs.

[0078] The step S12 specifically comprises: assigning a unique particle identifier to each physical particle, labeling the particle identifier to the particle sample images in the reflection light mode and the transmission light mode of the physical particle extracted in step S11, the particle identifier being the actual result (i.e. the true label) of particle classification, and the matched particle sample images in the reflection light mode and the transmission light mode being data pairs; subsequently, applying a preset data enhancement transformation method (including random horizontal flip, small-angle rotation, brightness and contrast fine-tuning, etc.) to the particle sample images in the reflection light mode and the transmission light mode, respectively, and performing normalization processing to adjust the pixel values thereof to a standardized numerical range suitable for neural network input, thereby improving the generalization ability and robustness of the model to various changes.

[0079] Since the images in two modes are intentionally collected for the same particle when collecting samples, the particle identifier of the specified physical particle can be labeled at the same time. The present application assigns a unique particle identifier to this physical particle. This particle identifier is its "true particle label", which is the actual result of particle classification.

[0080] Step S2: constructing a neural network model with double-flow input and shared backbone, the neural network model being set to receive particle sample images in the reflection light mode and the transmission light mode and output corresponding particle identifiers and high-dimensional feature vectors;

[0081] Reference Figure 2The neural network model of the application adopts a "dual-flow input, shared backbone" structure. In the embodiment, the neural network model is constructed based on a pre-trained EfficientNet-V2-S neural network. The pre-training here refers to the training of the EfficientNet-V2-S neural network on a large-scale, general image dataset (ImageNet dataset, containing millions of pictures and thousands of categories of daily objects), aiming to enable the network to learn general image feature extraction capability, which is usually completed by model developers. In the embodiment, the pre-trained EfficientNet-V2-S neural network model directly uses the publicly released model weights trained on the ImageNet dataset.

[0082] It should be noted that the optimization of model parameters by the composite objective function below is training on the specific dataset (microplastic images) of the application. The purpose is to fine-tune the model parameters using the composite objective function specially designed for this task based on the pre-trained model, so that it adapts to the specific task of particle matching. The application loads the pre-trained model weights as initial parameters, and then further trains and optimizes them using the data and composite objective function of the application.

[0083] According to the prior art document

Tan M, Le Q V. EfficientNetV2: Smaller Models and Faster Training[J]. 2021. DOI:10.48550 / arXiv.2104.00298

[0084] The neural network model comprises a dual-flow input module, a shared backbone network 2c, a feature processing module 2d and an output module 2e connected in sequence, and the dual-flow input module comprises two independent initial feature extraction modules.

[0085] Two independent initial feature extraction modules (i.e., the reflected light input module 2a and the transmitted light input module 2b) are used as a double-flow input module. The double-flow input module includes two independent initial feature extraction modules that do not share parameters at the initial stage, which are respectively used to process the particle sample images of the input reflected light mode and the transmitted light mode, and output the feature maps of the two initial feature extraction modules and stack them in the batch dimension. Each initial feature extraction module is composed of an initial feature extraction module of the EfficientNet-V2-S model, and the initial feature extraction module refers to the initial Stem layer, which is composed of a 3x3 convolution layer (Conv), a batch normalization layer (BatchNorm), and a SiLU activation function. The initial parameters of the two initial feature extraction modules are not shared to ensure differentiated learning of the characteristics of different modes.

[0086] The shared backbone network 2c includes a parameter-shared remaining feature extraction module, which is configured to receive the stacked feature maps and extract a three-dimensional feature map using the shared feature extraction module. The remaining feature extraction module. In the EfficientNet-V2-S neural network model, the remaining feature extraction module refers to the remaining stacked Fused-MBConv module layer and MBConv module layer, i.e., the Stage1-6 part.

[0087] The feature processing module 2d includes an adaptive average pooling layer and a batch normalization layer, which are used to receive the three-dimensional feature map and convert it into a high-dimensional feature vector. Thus, the three-dimensional feature map output by the shared backbone network is first compressed into a fixed-length one-dimensional feature vector through an adaptive average pooling layer, and then the one-dimensional vector is processed through a batch normalization layer; the feature processing module 2d realizes the feature compression and normalization of the three-dimensional feature map to obtain a high-dimensional feature vector.

[0088] The output module 2e includes a particle classification output branch for the training stage, a high-dimensional feature output branch, and a high-dimensional feature output branch for the inference stage, and each output branch generates a final output according to the high-dimensional feature vector.

[0089] In the training stage, to simultaneously achieve the training objectives of “particle classification discrimination” and “cross-modal feature alignment”, the output module 2e uses the particle classification output branch for the training stage and the high-dimensional feature output branch and outputs them simultaneously.

[0090] The particle classification output branch for the training stage is configured to input the high-dimensional feature vector into a linear classifier to output the prediction result of the particle classification, which is used to calculate the sample discrimination loss.

[0091] In the process of data preparation in step S1, a unique particle ID is assigned to each physical particle, and the particle ID is labeled to the particle sample image in reflection light mode and the particle sample image in transmission light mode. This particle ID is the actual result of particle classification. Sample discrimination loss is calculated using the standard Cross-Entropy Loss function, whose formula is:

[0092] ,

[0093] wherein, is the total number of samples in a batch, i is the sample serial number, K is the total number of particle classification categories, and k is the category serial number of particle classification, is the one-hot encoding of the actual result of particle classification (the position corresponding to the true ID is 1, and the rest is 0), is the probability that the k-th sample in the prediction result of particle classification belongs to the category k, which is usually the result of the Softmax function normalization after the classifier output.

[0094] The core use of the prediction result of particle classification is to provide a sample discrimination loss for the neural network model as a supervision signal in the training stage. By forcing the neural network model to correctly classify the particle ID of each physical particle, the neural network model can learn to distinguish between different physical particles and learn discriminative features. In the inference stage, the prediction result of particle classification is usually not directly used.

[0095] The high-dimensional feature output branch used in the training stage is set as follows: the high-dimensional feature vector from the feature processing module 2d is directly retained as a high-dimensional feature vector, which is used to calculate the cross-modal alignment measure loss. That is, the high-dimensional feature vector output from the feature processing module 2d is not passed through a linear layer classifier, but is directly used to calculate the cross-modal alignment measure loss. In terms of network structure, it is a high-dimensional feature output branch.

[0096] The goal of the entire training process is to learn a high-quality feature space, so that in this space, the high-dimensional feature vectors of particles from the same particle ID (regardless of the modality) are close to each other, and the high-dimensional feature vectors of particles from different IDs are far away from each other. The cross-modal alignment measure loss is used to measure how far the high-dimensional feature vectors output by the current model are from this “ideal state”.

[0097] In the present embodiment, the cross-modal alignment measure loss includes a cross-modal center loss and a center triplet loss L c ; the cross-modal center loss ​For realizing cross-modal intra-class alignment, center triplet loss L ct For realizing cross-modal intra-class alignment, center triplet loss L

[0098] For realizing cross-modal intra-class alignment, center triplet loss L cmc For realizing cross-modal intra-class alignment, center triplet loss L cmc is the average Euclidean distance between each sample and the feature center of its cross-modal same class. The cross-modal center loss L cmc , specifically comprising:

[0099] First, the feature center of each class under each modality is calculated:

[0100] (1)

[0101] Where i, j are sample serial numbers, is the feature center of the class with class serial number of particle classification under modality A, is the feature center of the class with class serial number of particle classification under modality B, denotes the feature vector of the i-th sample from modality A; denotes the feature vector of the j-th sample from modality B; denotes the class serial number of the k-th sample of particle classification, which is obtained from the actual result of particle classification; k is the class serial number of particle classification; denotes the feature center of all samples with class serial number of particle classification in modality A; denotes the feature center of all samples with class serial number of particle classification in modality B. In this embodiment, the modality can be the reflected light modality under optical microscope and the transmitted light modality under infrared spectrometer. Then, the average distance of all samples to the feature center of its cross-modal same class is calculated as the cross-modal center loss L

[0102] :

[0103] (2)

[0104] (2)

[0105] Where, , respectively represent the number of samples of modality A and modality B, is the total number of samples, denotes the i-th sample from modality A; denotes the j-th sample from modality B.​a class number of the particle classification of the i-th sample, a feature vector representing the i-th sample from modality A; a feature vector representing the i-th sample from modality B; a feature center of the same class of modality B representing the i-th sample from modality A, a feature center of the same class of modality A representing the i-th sample from modality B. center triplet loss L ct center triplet loss L ct is used to realize the inter-class separation, that is, to push away the features of different particles. It is used to punish the case that the distance between the sample and the "non-matching class center" is too close. center triplet loss L ct , specifically comprising:

[0106] First, the global feature center of the k-th class is calculated :

[0107]

[0108] (3)

[0109] (3)

[0110] where i is a sample number, a class number of the particle classification of the i-th sample, which is obtained from the actual result of the particle classification; k is a class number of the particle classification; is a feature vector of the i-th sample.

[0111] Then, the distance of each sample from all non-matching class centers is calculated as the center triplet loss :

[0112] (4)

[0113] where i is a sample number, a class number of the particle classification of the i-th sample, which is obtained from the actual result of the particle classification; is the total number of samples in a batch, K is the total number of classes of particle classification, and m is a set boundary value; is a feature vector of the i-th sample; k is a class number of the particle classification; is the global feature center of the k-th class.

[0114] ​​​​​​In the inference stage, the output module 2e only includes a high-dimensional feature output branch for the matching in the inference stage, and the high-dimensional feature output branch for the matching in the inference stage is configured to send the high-dimensional feature vector from the feature processing module to an L2 norm normalization layer for processing to generate a final high-dimensional feature matching vector for the matching. The L2 norm normalization is a fixed mathematical operation, and the function thereof is to scale the length (module) of a vector to 1 without changing the direction thereof. The role thereof is to map all feature vectors to a unit hypersphere, and this facilitates subsequent use of cosine similarity (i.e., vector dot product) for efficient similarity measurement.

[0115] Therefore, the neural network model constructed in the application can make the trained neural network model output consistent high-dimensional feature vectors according to the matched particle sample images of the reflection light mode and the transmission light mode by sharing the backbone network 2c and the corresponding loss function. The mechanism of sharing the backbone network forces the model to learn a unified, mode-independent high-dimensional feature space, ensuring that the images of the two modes follow the same conversion rule when deep feature extraction is performed. The loss function directly supervises the result. When it identifies that the two feature vectors are from the cross-modal image pair of the same particle, if the distance thereof is too far, the loss function will generate a penalty signal, and the network parameters are adjusted through the back propagation algorithm to shorten the distance. Conversely, when the two feature vectors are from different particles, if the distance is too close, the loss function will also generate a penalty signal, forcing the network to push them apart. By adjusting the network parameters through the back propagation, the model is forced to learn a feature representation that "eliminates the modal difference and focuses on the essence of the particle", and finally the high-dimensional feature vectors of the particles of the same ID are very close in the feature space regardless of which modal image is input.

[0116] Step S3: training the neural network model by using the particle sample images of the two modes constituting the data pair to obtain a matching model;

[0117] From the structure of the neural network model, whether an image of reflected light or an image of transmission light is input, the model will output a corresponding high-dimensional feature vector after forward propagation. From the perspective of model training, the application will combine multiple images of reflected light and images of transmission light constituting the data pair into the same "training batch" to train the neural network model. In order to let the model learn "matching", the images of the two modes must be included in the batch at the same time, so that the loss function can be effectively calculated and compared. Therefore, during training, the input must include particle sample images of the two modes. In this embodiment, a training batch must include multiple groups of "reflection light mode-transmission light mode" data pairs with the same particle identification.

[0118] As Figure 3As shown, in the step S3, the sample discrimination loss and the cross-modal alignment measurement loss are taken as a composite objective function, and the parameters of the neural network model are optimized through the composite objective function, so as to realize effective alignment of the cross-modal features.

[0119] The step S3 specifically includes:

[0120] Step S31: Structured data sampling is performed on the particle sample images of the two modalities constituting the data pair to obtain structured batch data.

[0121] Step S31 first adopts a PK sampling strategy to construct the training batch data. In each iteration, P different particle identifiers (i.e., particle classification categories) are first selected from the total training set, and then K particle sample images are randomly sampled for each modality of each particle identifier, thereby forming a structured batch with a size of P×K.

[0122] Specifically, for each particle identifier, the sampler collects K images from the particle sample images of its original reflection light modality and transmission light modality respectively and performs random image enhancement on the images, and the enhanced images collectively constitute the samples of the particle identifier in the training batch. In this process, the images of different modalities are always independent.

[0123] Step S32: The structured batch data is input into the neural network model for forward propagation, and the prediction result of the particle classification and the high-dimensional feature vector are synchronously output.

[0124] Step S33: The sample discrimination loss and the cross-modal alignment measurement loss are determined according to the prediction result of the particle classification and the high-dimensional feature vector, and are taken as a composite objective function.

[0125] Step S33 quantitatively evaluates the results of the forward propagation through the composite objective function.

[0126] The composite objective function includes the loss functions of the sample discrimination loss and the cross-modal alignment measurement loss. The sample discrimination loss acts on the particle classification prediction result of the neural network model. The loss component acts on the high-dimensional feature vector of the neural network model.

[0127] In calculating the sample discrimination loss, the prediction result of the particle classification and the actual result of the particle classification of the sample are input into a cross-entropy loss function for calculation.

[0128] In calculating the cross-modal alignment measurement loss, the cross-modal alignment measurement loss includes two sub-targets of cross-modal intra-class alignment (minimizing the distance between the sample feature and the corresponding particle complementary modality feature center) and inter-class separation (maximizing the distance between different particle feature centers), and finally obtains the cross-modal alignment measurement loss value.

[0129] The specific calculation formula refers to the sample discrimination loss provided in step S2 , the cross-modal center loss , and the center triplet loss L ct .

[0130] Step S34: Linearly combine the sample discrimination loss and the cross-modal alignment measure loss by a preset weight coefficient to form a total objective function of a single iteration ;

[0131] The sample discrimination loss , the cross-modal center loss , and the center triplet loss L ct refer to the calculation formulas provided in the previous section, and are:

[0132] (4)

[0133] (5)

[0134] (6)

[0135] (7)

[0136] (8)

[0137] The total objective function is:

[0138] (9)

[0139] wherein, , , is a preset weight coefficient for balancing different loss terms; is the sample discrimination loss, is the cross-modal center loss, and L ct is the center triplet loss.

[0140] Step S35: Adjust the parameters of the neural network model based on the total objective function.

[0141] Wherein, based on the total objective function, the gradient of each layer of the neural network model with respect to the trainable parameters is calculated by the back propagation algorithm, and the optimizer updates the parameters of the neural network model according to the gradient.

[0142] Step S36: Repeat steps S31 to S35 to perform iterative training until the convergence judgment condition is met, at which time the matching model is obtained.

[0143] Wherein, repeat steps S31 to S35, and perform multiple rounds of iterative training on the entire training data set until the convergence judgment condition is met. The convergence judgment condition includes that the performance indicator of the neural network model reaches saturation on an independent validation set, or meets other preset convergence conditions.

[0144] When the training stops, save the current parameters of the neural network model as the matching model.

[0145] Step S4: In the inference stage, use the matching model to perform cross-modal matching on the query image (Query) and the gallery image (Gallery) to obtain the final matching result. Thus, model inference and semi-automatic matching are realized.

[0146] Wherein, the query image (Query) is a microplastic image of one modality. When the query image is a microplastic image of the reflection light modality, the gallery image is a set of all microplastic images of the transmission light modality to be matched. Conversely, when the query image is a microplastic image of the transmission light modality, the gallery image is a set of all microplastic images of the reflection light modality to be matched. The query image is usually one, and the number of gallery images can be any number, depending on the size of the candidate gallery. In this embodiment, considering the device type, a single transmission light modality image is used as the query image, and a reflection light modality image is used as the gallery image.

[0147] The step S4 further includes combining automated calculation with manual review to ensure result reliability.

[0148] As shown in Figure 4 The step S4 specifically includes:

[0149] Step S41: Process the query image (Query) and the gallery image (Gallery) to obtain particle sample images of the two modalities, and use the matching model to extract features to obtain high-dimensional feature vectors for matching of the query image and the gallery image as query feature vectors and gallery feature matrix;

[0150] For the query image (Query) and the gallery image (Gallery), when they contain multiple microplastic particles, respectively perform the extraction of the particle position consistent with step S11. Load the matching model and switch to the inference mode, and send the particle sample images of the two modalities into the matching model respectively, extract the high-dimensional feature vectors for matching of each image through forward propagation, and integrate to obtain one query feature vector and one gallery feature matrix composed of all gallery image feature vectors.

[0151] It should be noted that microplastic images can be fed into the neural network model individually. During the training phase, the model takes a "batch" of microplastic images containing multiple reflected and transmitted light modes as input. However, each microplastic image in the batch is independently forward-propagated through the network, and the corresponding particle prediction and high-dimensional feature vector are output through the training phase branch. During the inference phase, the neural model feeds the image into the corresponding preliminary feature extraction module and subsequent parts of the model according to the specified mode, and extracts the high-dimensional feature vector for matching after forward propagation.

[0152] Specifically, if the library contains N images, and each feature vector is D-dimensional, then the library feature matrix is ​​an N×D matrix. Each row of the library feature matrix represents a high-dimensional feature vector of a library image, normalized by L2.

[0153] As mentioned above, L2 normalization is a mathematical operation that scales the length (magnitude) of a vector to 1 without changing its direction. Among these, high-dimensional feature vectors... It can be represented as:

[0154] (10)

[0155] Where D is the dimension of the vector, which is 1280 in this embodiment, and i is the ordinal number of the dimension. , It is the component of the vector in each dimension.

[0156] vector The L2 norm of a vector is the square root of the sum of the squares of all its components. (High-dimensional eigenvectors) L2 norm The calculation formula is:

[0157] (11)

[0158] Where i is the ordinal number of the dimension, and D is the dimension of the vector. , It is the component of the vector in each dimension.

[0159] High-dimensional feature vectors used for matching (That is, the high-dimensional feature vector after L2 normalization) can be expressed as:

[0160] (12)

[0161] Where i is the ordinal number of the dimension, and D is the dimension of the vector. , It is the component of the vector in each dimension. High-dimensional feature vectors The L2 norm.

[0162] The library feature matrix is composed of a set of high-dimensional feature vectors for matching of all individual library images. Each row of the matrix represents a high-dimensional feature vector for matching of a library image.

[0163] If there are N library images in the library, each image has a high-dimensional feature vector of D dimensions (D = 1280 in this embodiment), and the high-dimensional feature vector for matching of the i-th library image is which can be expressed as:

[0164]

[0165] wherein, is the D-th dimension component of the high-dimensional feature vector for matching of the i-th library image.

[0166] The library feature matrix G composed of N such feature vectors can be expressed as:

[0167] (13)

[0168] Step S42: According to the query feature vector and the library feature matrix, similarity calculation and threshold screening are performed to obtain effective candidate matching items of the image to be queried.

[0169] In this embodiment, through matrix multiplication of the query feature vector and the library feature matrix, the cosine similarity between the granular sample images of all images to be queried and library images is efficiently calculated, and a similarity score matrix is generated. Each similarity score in the similarity score matrix is compared with a preset similarity threshold T, and the library image with a similarity score higher than the similarity threshold T is regarded as an effective candidate matching item, and the library image with a similarity score lower than T is directly determined as not matching and eliminated.

[0170] The preset similarity threshold T is preset according to an empirical value and can be flexibly adjusted. In this embodiment, testing is performed on an independent test set data, and by analyzing the scores of different modal image data pairs of the same granular identifier and the scores of different modal image data pairs of different granular identifiers, a similarity threshold is selected. In actual application, it can be flexibly adjusted according to specific circumstances to achieve the best effect.

[0171] In this embodiment, the dimension q of the feature vector extracted by the EfficientNet-V2-S neural network used is 1280. The query feature vector q is an L2 normalized 1x1280 row vector. It represents the features of a single image to be queried.

[0172] (14)

[0173] The library feature matrix G: an L2-normalized N x 1280 matrix, where N is the total number of images in the library. Each row of the library feature matrix G is a high-dimensional feature vector for matching a library image, and is given by:

[0174] (15)

[0175] The similarity score matrix S is obtained by transposing the library feature matrix G and then multiplying it with the query vector q. Since all the feature vectors (the query feature vector q and each row in the library feature matrix G) are L2-normalized, the dot product (inner product) between them is equal to their cosine similarity. The resulting similarity score matrix S is a 1 x N row vector. The first similarity score in the similarity score matrix S is the similarity score between the query feature vector q and the first library image feature vector, the second similarity score in the similarity score matrix S is the similarity score between the query feature vector q and the second library image feature vector, and so on.

[0176] The similarity score matrix S is given by:

[0177] (15)

[0178] where s1 is the first similarity score in the similarity score matrix, s2is the second similarity score in the similarity score matrix, and S is the similarity score matrix, q is the query feature vector, and G is the library feature matrix.

[0179] Step S43: determining the final matching result according to the valid candidate matching item;

[0180] The step S43 specifically includes:

[0181] According to the number of valid candidate matching items, different decision paths are executed:

[0182] When the number of valid candidate matching items = 1: it is determined that the matching result is clear and the confidence is high, and the valid candidate matching item is output as the final matching result;

[0183] When the number of valid candidate matching items > 1: the determination result is not unique, manual intervention is required, and all valid candidate matching items are sorted in descending order of similarity scores, and manual review is performed to determine whether to output the final matching result or output no matching result. Specifically, the top N candidate matching items and their corresponding similarity scores are presented on the human-computer interaction interface for the operator to confirm and determine; the operator can select one of the library images as the final matching result, or all reject to output no matching result.

[0184] When the number of valid candidate matching items = 0: no matching item is determined, and no matching result is output.

[0185] As described above, in the present application, the total loss (i.e., the composite objective function) includes a sample discrimination loss (i.e., a basic classification loss) + a cross-modal alignment measure loss (i.e., an auxiliary loss). In the present embodiment, the cross-modal alignment measure loss includes a cross-modal center loss (a variant of the center loss) and a center triplet loss L ct (a variant of the triplet loss); the cross-modal center loss is used to achieve cross-modal intra-class alignment, and the center triplet loss L ct is used to achieve inter-class separation. The cross-modal center loss ct and the center triplet loss L ct together serve as a composite cross-modal alignment measure loss, which is used together with the sample discrimination loss (cross-entropy loss) to ultimately achieve intra-class alignment and inter-class separation. However, in other embodiments, this composite cross-modal alignment measure loss can be replaced by other advanced measure loss functions that can be used to achieve both cross-modal intra-class alignment and inter-class separation, or at least intra-class alignment, to further optimize the distance between samples based on the basic classification loss. The cross-modal alignment measure loss includes, but is not limited to, at least one of contrastive loss (Contrastive Loss), triplet loss (Triplet Loss), N-pair loss (N-Pair Loss), multi-similarity loss (Multi-Similarity Loss), or circle loss (Circle Loss), center loss (Center Loss), ring loss (Ring Loss), etc.

[0186] In other embodiments, the replacement of the neural network backbone network: the "dual-flow input, shared backbone" neural network model adopted by the present application is not limited to the EfficientNet network model, and any neural network model including an initial feature extraction module and a remaining feature extraction module can be appropriately modified (construct dual-flow input and change the output layer to a modified feature processing module and an output module) to obtain a "dual-flow sharing" design that partially shares parameters and partially shares parameters. The initial feature extraction module and the remaining feature extraction module of the neural network model can be replaced by any other advanced feature extraction network, including but not limited to the Residual Network (ResNet) series: such as ResNet-50, ResNet-101; Vision Transformer (ViT) and its variants: such as Swin Transformer, DeiT; Dense Connection Network (DenseNet) series: such as DenseNet-121; lightweight network series: such as MobileNetV3, ShuffleNet V2, etc.

[0187] In other embodiments, the extraction method of the particle position can also be replaced as needed. For example, in the data preprocessing stage, the dynamic threshold segmentation method used to extract the particle region of interest (ROI) can also be replaced by other image segmentation techniques, such as adaptive threshold segmentation, region growing algorithm, or using an independent semantic segmentation network (such as U-Net) for higher precision pre-segmentation.

[0188] Referring to Figure 5 The present application also provides a cross-modal matching system applied to microplastic image recognition, which comprises:

[0189] A particle extraction module is configured to: in the training stage and the inference stage, extract the particle position of a microplastic image containing multiple microplastic particles to obtain a particle sample image, and directly use a microplastic image of a single microplastic particle as a particle sample image; and in the training stage, construct data pairs according to the particle sample images of the two modalities to obtain the particle sample images of the two modalities constituting the data pairs;

[0190] A model training module is configured to: train a neural network model using the particle sample images of the two modalities constituting the data pairs to obtain a matching model; the neural network model is a dual-flow input, shared backbone neural network model configured to receive particle sample images of the reflection light modality and the transmission light modality and output corresponding particle labels and high-dimensional feature vectors;

[0191] The feature extraction and semi-automatic matching module is configured to perform cross-modal matching on the query image and the gallery image using the matching model in the inference stage to obtain a final matching result.

[0192] Specifically, the feature extraction and semi-automatic matching module loads the matching model trained by the model training module, receives the particle sample images of the two modalities obtained by the particle extraction module in the inference stage according to the query image and the gallery image, and performs feature extraction using the matching model to obtain high-dimensional feature vectors of the query image and the gallery image for matching, as the query feature vector and the gallery feature matrix. Then, similarity calculation and threshold screening are performed according to the query feature vector and the gallery feature matrix to obtain effective candidate matching items of the query image. Finally, the final matching result is determined according to the effective candidate matching items.

[0193] The cross-modal sample matching method applied to microplastic image recognition has the following advantages compared with other matching algorithms:

[0194] 1. Breakthrough in applicability: from clear contours to blurred boundaries, effectively handling blurred and cross-modal images that traditional algorithms cannot handle.

[0195] Specifically, traditional algorithms such as contour matching and feature point matching are highly dependent on clear and stable geometric features in the image, and are limited to analyzing and comparing the two-dimensional outer boundary contours of particles, extracting a low-dimensional descriptor based on geometric morphology. For transmission light images with blurred edges and no explicit physical boundaries due to imaging principles, traditional algorithms will completely fail to find reliable matching basis.

[0196] The present application learns and extracts a high-dimensional abstract feature vector from all pixel data in the particle region of interest through a deep neural network. Unlike traditional algorithms, the matching model of the present application does not rely solely on clear geometric boundaries or local textures, but establishes a feature model by learning the overall distribution pattern of pixels. Therefore, the present application can encode complex discriminative information representing the intrinsic properties of particles and effectively handle blurred images, thereby solving the applicability limitations of traditional matching algorithms in cross-modal applications.

[0197] 2. Innovation of matching paradigm: adaptive metric learning space is used instead of fixed rule matching, achieving more accurate and efficient cross-modal feature alignment.

[0198] The optimization goal of conventional algorithms is general, and this technology relies on fixed, artificially designed mathematical formulas to determine similarity, with static matching rules that cannot be optimized for specific tasks and are not specifically designed for the "modal gap" caused by different imaging principles.

[0199] The present application adopts a learnable cross-modal alignment training strategy, which can automatically and adaptively optimize a feature metric space. In this space, different modal features from the same particle are mapped to close positions, while the features of different particles are separated from each other. This data-driven and dynamically optimized matching paradigm provides an effective and stable supervision signal for modal differences, enabling the model to more efficiently learn an effective feature space specifically designed to eliminate modal differences, thereby improving matching accuracy compared to conventional deep learning methods.

[0200] 3. Integrity of the method flow: the overall robustness of the system is improved through end-to-end design.

[0201] Compared with a single matching algorithm: whether it is a traditional algorithm or a conventional deep learning algorithm, it is usually only an independent calculation module.

[0202] Advantages of the present application: the present application is a complete end-to-end technical solution including specific field preprocessing. It first preprocesses and separates the input data through a special particle extraction module, eliminating the interference of irrelevant information such as background. This systematic design of combining the particle extraction module with the matching algorithm ensures that every step from input to output contributes to the final matching, making the overall robustness and performance in real complex environments of the system more robust than single and isolated matching algorithms.

[0203] In summary, the existing matching algorithms have various differences in the results of spectral instruments and microscope shooting, including but not limited to differences in picture clarity, changes in particle position, and background differences caused by different light sources, etc., which make it impossible to directly match the results of spectral instruments and microscope shooting. The method of the present application realizes the matching of high-dimensional feature vectors of two groups of images of different modalities through a double-flow input, shared backbone neural network model, realizes the matching of microplastic images shot by different modalities, and thus can provide accurate material species (spectral image converted into text label) and clear enough images to the image recognition model for training; after the two groups of data sets shot by spectral instruments and microscopes are matched by the cross-modal matching method of the present application, they can be further used as input parameters of the image recognition model to realize accurate identification of microplastics.

[0204] It should be understood that the above embodiments are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-modal sample matching method applied to microplastic image recognition, characterized in that, The method comprises the following steps: Step S1: in the training phase, the microplastic image is processed to obtain a particle sample image; and data pairs are constructed according to the particle sample images of the two modalities, so as to obtain the particle sample images of the two modalities constituting the data pairs; Step S2: a neural network model with double-flow input and shared backbone is constructed, and the neural network model is set to receive the particle sample images of the reflection light modality and the transmission light modality and output corresponding particle identification and high-dimensional feature vectors; Step S3: the neural network model is trained by using the particle sample images of the two modalities constituting the data pairs, so as to obtain a matching model; Step S4: in the inference phase, the matching model is used to perform cross-modality matching on a to-be-queried image and a library image, so as to obtain a final matching result.

2. The cross-modal sample matching method for microplastic image recognition according to claim 1, wherein, The step S1 specifically comprises: Step S11: the particle position of a microplastic image containing a plurality of microplastic particles is extracted, and a single microplastic particle image is directly taken as a particle sample image; Step S12: the particle sample images of the reflection light modality and the transmission light modality are constructed and enhanced to obtain the particle sample images of the reflection light modality and the transmission light modality as data pairs; The step S11 comprises: Step S101: the input microplastic image is converted into a single-channel grayscale image; Step S102: filtering processing is performed on the single-channel grayscale image to suppress noise and smooth the image; Step S103: image segmentation is performed on the filtered microplastic image to obtain a preliminary binary mask; Step S104: a contour finding algorithm is performed on the preliminary binary mask to identify the contours of all independent and closed foreground regions; Step S105: area screening is performed on the identified foreground region contours, and micro contours with an area smaller than a preset area threshold are removed, and only the remaining contours are reserved as effective contours of real particles; Step S106: a final binary mask is generated according to the effective contours of the real particles, and a particle sample image is extracted therefrom.

3. The cross-modal sample matching method for microplastic image recognition according to claim 2, characterized in that, In the step S103, the image segmentation adopts one of a dynamic threshold segmentation method, an adaptive threshold segmentation, a region growing algorithm, and an independent semantic segmentation network; Step S106 specifically comprises: filling the effective contours of the real particles and drawing them to a blank canvas to form a final binary mask; calculating the minimum bounding rectangle of the effective contours of the real particles to obtain boundary box coordinates; using the final binary mask and the boundary box coordinates, two types of training samples are extracted from the original colored microplastic image as particle sample images: ① performing a bitwise AND operation using the final binary mask to extract a particle region of interest image completely shielded by the background, so as to extract information corresponding to the effective contour position; ② using the boundary box coordinates to crop a rectangular region corresponding to the boundary box to obtain a boundary box image containing the particle and part of the adjacent background.

4. The cross-modal sample matching method for microplastic image recognition according to claim 1, wherein, In the step S2, the neural network model comprises: a double-flow input module, a shared backbone network, a feature processing module and an output module connected in sequence, the double-flow input module comprises two initial feature extraction modules independent of each other; the shared backbone network comprises a remaining feature extraction module sharing parameters; the feature processing module comprises an adaptive average pooling layer and a batch normalization layer; and the output module comprises a particle classification output branch and a high-dimensional feature output branch for the training stage, and a high-dimensional feature output branch for the inference stage.

5. The cross-modal sample matching method for microplastic image recognition according to claim 4, characterized in that, The neural network model comprises an initial feature extraction module and a remaining feature extraction module; the initial feature extraction module and the remaining feature extraction module are one of the following except for the initial feature extraction modules of the EfficientNet network model, the residual network series, the visual attention transformer, the densely connected network series and the lightweight network series.

6. The cross-modal sample matching method for microplastic image recognition according to claim 4, wherein, The particle classification output branch for the training stage is configured to input the high-dimensional feature vector into a linear classifier to output a prediction result of particle classification, and is used for calculating a sample discrimination loss; The high-dimensional feature output branch for the training stage is configured to directly retain the high-dimensional feature vector from the feature processing module as a high-dimensional feature vector, and is used for calculating a cross-modal alignment measurement loss; The high-dimensional feature output branch for the inference stage is configured to input the high-dimensional feature vector from the feature processing module into an L2 norm normalization layer for processing to generate a final high-dimensional feature matching vector for matching; The sample discrimination loss is calculated using a standard cross-entropy loss function. The cross-modal alignment measurement loss comprises at least one of a contrast loss, a triplet loss, an N-pair loss, a multi-similarity loss, a circle loss, a center loss and a ring loss.

7. The cross-modal sample matching method for microplastic image recognition according to claim 6, characterized in that, In the step S3, the sample discrimination loss and the cross-modal alignment measurement loss are taken as a composite objective function, and the parameters of the neural network model are optimized through the composite objective function to realize effective alignment of cross-modal features.

8. The cross-modal sample matching method for microplastic image recognition according to claim 7, characterized in that, The step S3 specifically comprises: Step S31: performing structured data sampling on the particle sample images of the two modalities constituting the data pair to obtain structured batch data; Step S32: inputting the structured batch data into the neural network model for forward propagation to synchronously output a prediction result of particle classification and a high-dimensional feature vector; Step S33: determining the sample discrimination loss and the cross-modal alignment measurement loss according to the prediction result of particle classification and the high-dimensional feature vector to serve as a composite objective function; Step S34: linearly combine the sample discrimination loss and the cross-modal alignment measure loss according to a preset weight coefficient to form a total objective function of a single iteration ; Step S35: adjusting the parameters of the neural network model based on the total objective function; Step S36: repeating the steps S31 to S35 to perform iterative training until a convergence judgment condition is met, and a matching model is obtained at this time.

9. The cross-modal sample matching method for microplastic image recognition according to claim 1, wherein, The step S4 specifically comprises: Step S41: processing the to-be-queried image and the library image to obtain particle sample images of the two modalities, and performing feature extraction on the particle sample images by using the matching model to obtain a high-dimensional feature vector for matching of the to-be-queried image and the library image as a query feature vector and a library feature matrix; Step S42: performing similarity calculation and threshold screening according to the query feature vector and the library feature matrix to obtain an effective candidate matching item of the to-be-queried image. Step S43: determining a final matching result according to the valid candidate matching items.

10. A cross-modal matching system applied to microplastic image recognition, comprising: a particle extraction module configured to, in a training phase and an inference phase, extract particle positions for microplastic images containing multiple microplastic particles to obtain particle sample images, and directly use microplastic images of single microplastic particles as particle sample images, and in the training phase, construct data pairs according to particle sample images of two modalities to obtain particle sample images of the two modalities constituting the data pairs; a model training module configured to train a neural network model using particle sample images of two modalities constituting data pairs to obtain a matching model, wherein the neural network model is a neural network model with double-stream input and shared backbone, configured to receive particle sample images of a reflection light modality and a transmission light modality and output corresponding particle labels and high-dimensional feature vectors; a feature extraction and semi-automatic matching module configured to, in the inference phase, use the matching model to perform cross-modal matching on a to-be-queried image and a library image to obtain a final matching result.

Images