Microplastic species identification method based on concept constraint multi-task instance segmentation

Abstract

The application provides a microplastic species identification method based on concept constraint multi-task instance segmentation, comprising: performing instance-level labeling and concept label labeling on a microscopic image containing microplastic particles to obtain an input sample; constructing a concept constraint multi-task instance segmentation network with microplastic particle instance segmentation and polymer category identification as a main task and prediction of a concept label as a secondary task; the network processes the output result of the main task according to the output result of the secondary task to obtain a single-concept additional feature area, avoids missing detection, generates a minimum value result according to the output result of the secondary task and the output result of the main task, and further obtains a final identification result; training the network; and inputting a to-be-identified microscopic image into the trained network to obtain the final identification result. The method of the application completes microplastic particle instance segmentation and species identification under the condition of explicitly learning and utilizing the concept attributes of microplastics.

Description

Technical Field

[0001] This invention belongs to the field of machine vision, specifically relating to a method for identifying microplastic species based on concept-constrained multi-task instance segmentation, which is implemented based on concept-constrained multi-task instance segmentation. Background Technology

[0002] Microplastics are widely found in water, soil and organisms, and their accurate identification and classification are key aspects of environmental monitoring.

[0003] Microplastic identification is essentially a complex small-target visual recognition problem. Most existing microplastic image recognition models employ direct training of target detection or classification networks, following a process of "input microscopic image—output polymer category or target box." For example, convolutional neural networks, Transformer detection networks, or instance segmentation networks are commonly used to locate and classify microplastic particles in microscopic images. Representative implementations include YOLO-based target detection, Mask R-CNN or other mask branch networks for instance segmentation, and Transformer-based end-to-end detection / segmentation schemes.

[0004] The aforementioned schemes generally employ a single primary task training logic, directly using class labels, bounding boxes, or masks as supervision signals to optimize the error between the model output and the labeled results. These schemes are simple to implement and easy to deploy. While these methods can achieve high fitting accuracy with a certain data scale, they typically assume the model will learn the optimal discriminative features on its own, thus having the following drawbacks: (1) Existing mainstream detection or segmentation models mainly rely on the main task labels for training, and the feature extraction process is black-boxed. Researchers cannot actively participate in the model's learning of key visual concepts, resulting in an uncontrollable focus of the model on microplastic features. Its feature learning process is basically in a black-box form. The model mainly relies on data-driven automatic extraction of discriminative information. Researchers find it difficult to explicitly specify which key visual features related to microplastic recognition the model should prioritize during training.

[0005] (2) Existing methods are usually only responsible for the final category and location, and do not include conceptual attributes that are highly related to microplastic identification, such as color, transparency, edge sharpness, aspect ratio, roundness, and texture roughness, in explicit supervision. Therefore, confusion is likely to occur between similar categories, and false negatives are likely to occur on weak response targets.

[0006] (3) Even if the conventional multi-task method introduces auxiliary attribute branches, it mostly stays at the level of "parallel prediction". It lacks technical mechanisms for enhancing the intersection of the output results of the main task branch 2b and the prediction results of the concept attributes, preserving the common area and screening for consistency. Therefore, it is difficult to systematically solve the two types of problems: "cannot be identified" and "cannot be removed if identified incorrectly".

[0007] (4) Target detection is expressed by bounding boxes, which makes it difficult to accurately depict pixel-level conceptual attributes such as transparency, local texture, edge details and irregular contours, which is not conducive to the fine recognition of complex morphologies of microplastics.

[0008] In practical applications, different polymer microplastics often exhibit similar colors, outlines, blurred boundaries, large variations in transparency, and varying degrees of surface weathering. When relying solely on the main task for supervision, the model is prone to two types of problems: First, some particles may clearly possess identifiable microplastic characteristics, but due to insufficient response from the main task, they may be missed or misclassified. Second, although some non-target regions or other polymer particles may be incorrectly activated by the main task, the lack of further screening based on conceptual attributes such as color, shape, transparency, and texture makes it difficult to effectively remove false detections and misclassifications.

[0009] General multi-task learning methods introduce auxiliary tasks in addition to the main task, such as simultaneously predicting category and certain morphological attributes, and improving feature representation by sharing a backbone network. However, conventional multi-task learning only optimizes multiple tasks in parallel at the loss function level, lacking mechanisms for explicit selection, fusion, and error suppression.

[0010] In existing solutions, conceptual attributes often exist only as additional labels or are only used for visual interpretation. There is no instance segmentation and recognition method for microplastic microscopic images with "explicit concept supervision - consistency analysis between concept and main task space - false detection suppression" as the core chain.

[0011] Therefore, it is necessary to provide a concept-constrained multi-task instance segmentation model for microplastic microscopic images, so as to explicitly learn and utilize conceptual attributes such as color, transparency, morphology, and texture to enhance and filter the output results of the main task branch 2b of microplastic particle instance segmentation and polymer category recognition. Summary of the Invention

[0012] The purpose of this invention is to provide a method for identifying microplastic species, so as to complete the segmentation and species identification of microplastic particles by explicitly learning and utilizing the conceptual attributes of microplastics.

[0013] To achieve the above objectives, this invention provides a method for identifying microplastic species based on concept-constrained multi-task instance segmentation, comprising: Step S1: Perform instance-level annotation and concept labeling on the microscopic images containing microplastic particles to obtain the input sample; Step S2: Construct a concept-constrained multi-task instance segmentation network with microplastic particle instance segmentation and polymer category identification as the main task and concept label prediction as the secondary task. The concept-constrained multi-task instance segmentation network processes the output of the main task based on the output of the secondary task to obtain additional feature regions for single concepts, avoiding missed detections. The minimum value result is generated by intersecting the output of the secondary task and the output of the main task, thus obtaining the final recognition result. Step S3: Train the concept-constrained multi-task instance segmentation network; Step S4: Input the microscopic image to be identified into the trained concept-constrained multi-task instance segmentation network to obtain the final recognition result.

[0014] The concept-constrained multi-task instance segmentation network as a whole includes a shared backbone feature extraction module, a main task branch and a secondary target branch group located at the output end of the backbone feature extraction module, a collaborative fusion module located at the output end of the main task branch and the secondary target branch group, a multi-concept intersection module located at the output end of the main task branch and the secondary target branch group, and an output module located at the output end of the collaborative fusion module and the multi-concept intersection module. The collaborative fusion module processes the output of the main task based on the output of the secondary task to obtain the additional feature region of a single concept, thus avoiding missed detections; the multi-concept intersection module generates the minimum value result by intersecting the output of the secondary task and the output of the main task.

[0015] The backbone feature extraction module receives a single microscopic image or a sub-block of a microscopic image as input, and is used to extract multi-scale features for use by the main task branch and the secondary target branch group. The main task branch employs a segmentation framework and outputs the main task's results, which include at least: particle bounding boxes, particle instance masks, polymer category scores, and main task confidence scores. The secondary task branch group outputs predictions for multiple concept attributes in parallel, serving as the secondary task's results. The concept attribute predictions include a concept response layer. ,in k is the ordinal number of the concept attribute, and n is the total number of concept attributes; The collaborative fusion module receives the granular instance mask output by the main task branch and the concept response layers output by the secondary target branch group, performs a one-to-one spatial comparison of single concepts, assigns a concept label to each concept, and generates corresponding single-concept overlapping regions. Single-concept additional feature region and spatial overlap The collaborative fusion module ultimately outputs concept labels and the additional feature regions of each concept label for each single concept. ; The multi-concept intersection module outputs each concept response layer based on the secondary target branch group. Calculate the intersection of all concept layers. And utilize it on the main area layer Processing is performed to generate the minimum result. Suspected missed detection areas and the main region without feature support region ; The output module receives the output results of the main task and the secondary task, and also receives the single-concept additional feature region output by the collaborative fusion module. And the minimum value result output by the multi-concept intersection module. Suspected missed detection areas and the main region without feature support region Output the final recognition result; The final identification result includes at least the maximum value result. and minimum result Statistical indicators and final identification parameters, including single-concept replenishment rate. Maximum expansion rate Minimum retention rate Suspected false negative rate and featureless support rate The final identification parameters include particle bounding boxes, particle instance masks, polymer category scores, main task confidence scores, and concept attribute prediction results.

[0016] The backbone feature extraction module uses the initial feature extraction block of Swin Transformer-Tiny and the Feature Pyramid Network (FPN); the main task branch uses the two-stage instance segmentation head structure of Mask R-CNN.

[0017] In the collaborative fusion module, The intersection of all concept layers for: , Minimum result for: , Suspected missed detection areas for: = , Main region without feature support region for: , in, For concept response layers, where k is the ordinal number of the concept attribute, n is the total number of concept attributes; M is the coating of the main region.

[0018] Maximum value result for: , in, Main area layer, For single-concept additional feature regions, Areas suspected of being missed during detection; , , , , , in, For single-concept replenishment rate, This represents the maximum expansion rate. Minimum retention rate This represents the suspected false negative rate. The support rate is without characteristics. To prevent small constants from being divided by zero.

[0019] Step S3 specifically includes: Step S301: Input the training samples into the concept-constrained multi-task instance segmentation network and perform multi-task forward propagation; Step S302: Calculate the main task loss; Step S303: Calculate the conceptual task loss; Step S304: Obtain the total training loss function based on the main task loss and the concept task loss; Step S305: Continuously calculate the total loss function and update the parameters of the concept-constrained multi-task instance segmentation network until the training termination condition is met, at which point the trained concept-constrained multi-task instance segmentation network is obtained.

[0020] In step S302, the instance mask loss is calculated based on the output of the main task branch and the instance-level annotation information. Polymer classification loss Weighted fusion yields the main task loss. ; In step S303, the conceptual task loss for: , in, For the first Prediction results for each concept For the first A concept label, For the first Each concept loss weight, This refers to the corresponding loss function. Loss functions include classification loss, regression loss, or pixel-level segmentation loss. In step S304, the main task loss is... and concept task loss The data is fused according to preset fusion weights to obtain the total training loss function. .

[0021] Step S1 specifically includes: Step S101: Acquire microscopic images containing microplastic particles and perform preprocessing on the original microscopic images; Step S102: Perform instance-level annotation on each microplastic particle in the microscopic image. The instance-level annotation information includes at least the instance mask and polymer category label. Step S103: Label the concepts; Step S104: Establish unified coding rules to encode different concept tags and obtain the coding results of concept tags.

[0022] The concept labels include multiple categories such as color category, transparency level, aspect ratio level, roundness level, texture roughness level, edge sharpness level, and surface defect level.

[0023] The microplastic species identification method of this invention sets up multiple conceptual tasks in addition to the main task, and establishes a collaborative fusion module and a multi-concept intersection module. On the basis of using the capabilities of the main task to complete microplastic particle instance segmentation and species identification, it further utilizes explicitly learned conceptual attributes of microplastics such as color, transparency, morphology, and texture to supervise the results of the main task. This supervision enhances the correct target response and filters out erroneous responses that do not conform to conceptual constraints. The output results of the main task branches of microplastic particle instance segmentation and polymer category identification are enhanced and filtered, thereby improving the accuracy of microplastic identification, reducing the false negative rate and false positive rate, and enhancing the interpretability and controllability of the model results. Attached Figure Description

[0024] Figure 1 This is a flowchart of the sample construction and concept annotation of the concept-constrained multi-task instance segmentation and recognition method for microplastic microscopic images according to the present invention; Figure 2 This is a block diagram of the overall structure of the concept-constrained multi-task instance segmentation network for the concept-constrained multi-task instance segmentation and recognition method for microplastic microscopic images of the present invention. Figure 3 This is a flowchart of the joint training of the concept-constrained multi-task instance segmentation and recognition method for microplastic microscopic images according to the present invention. Detailed Implementation

[0025] This invention discloses a concept-constrained multi-task instance segmentation and recognition method for microplastic microscopic images. The method takes microscopic images as input, with microplastic particle instance segmentation and polymer category recognition as the main tasks. Based on this, multiple concept task branches are introduced to explicitly model the conceptual attributes of microplastics, such as color, transparency, morphology, and texture. Subsequently, a collaborative fusion module and a multi-concept intersection module are used to perform secondary filtering and correction on the main task output to obtain the final microplastic type recognition result. The concept constraints of this invention refer to using concept attributes obtained through explicit annotation or rule quantization as supervisory signals to guide, filter, and suppress errors in the intermediate features and output results of the model.

[0026] like Figure 1 As shown, the concept-constrained multi-task instance segmentation and recognition method for microplastic microscopic images of the present invention includes: Step S1: Sample Construction and Concept Labeling: Perform instance-level annotation and concept labeling on the microscopic images containing microplastic particles to obtain the input samples; Step S1 specifically includes: Step S101: Acquire microscopic images containing microplastic particles and perform preprocessing on the original microscopic images; preprocessing includes cropping, scale unification, background normalization, color normalization, noise suppression, etc.

[0027] Step S102: Perform instance-level annotation on each microplastic particle in the microscopic image. The instance-level annotation information includes at least the instance mask and polymer category label. Instance-level annotation information may include bounding box information when necessary to accommodate the training requirements of different instance segmentation frameworks.

[0028] Step S103: Label the concepts; The concept labels can be obtained through manual interpretation, image rule extraction, semi-automatic statistics, or a combination of both. Concept labels include, but are not limited to, color category, transparency level, aspect ratio level, roundness level, texture roughness level, edge sharpness level, and surface defect level.

[0029] Step S104: Establish unified coding rules to encode different concept tags and obtain the coding results of concept tags.

[0030] For discrete concepts, classification label coding is used, while for continuous concepts, interval level coding or regression value coding is used.

[0031] In this embodiment, the concept label corresponds to the local space. Therefore, after step S104, step S105 is also included: the encoding result of the concept label is processed into the form of a concept mask or concept response region annotation to support pixel-level supervision.

[0032] Step S2: Construct a concept-constrained multi-task instance segmentation network with microplastic particle instance segmentation and polymer category identification as the main tasks and the prediction of the concept tags as the secondary task. Reference Figure 2 The concept-constrained multi-task instance segmentation network structure of this invention adopts an improved SwinTransformer+FPN+Mask R-CNN. Based on the original SwinTransformer+FPN+Mask R-CNN framework, this invention uses the existing localization branch, segmentation branch, and category prediction branch as the main task branch 2b. In addition to the main task branch 2b, a secondary target branch group 2c (which includes multiple parallel concept attribute prediction branches) is added. A collaborative fusion module 2d is added at the output of the main task branch 2b and the secondary target branch group 2c. A multi-concept intersection module 2e is added at the output of the secondary target branch group 2c and the collaborative fusion module 2d, thereby forming the concept-constrained multi-task instance segmentation network.

[0033] The concept-constrained multi-task instance segmentation network as a whole includes a shared backbone feature extraction module 2a, a main task branch 2b and a secondary target branch group 2c located at the output end of the backbone feature extraction module 2a, a collaborative fusion module 2d located at the output end of the main task branch 2b and the secondary target branch group 2c, a multi-concept intersection module 2e located at the output end of the main task branch 2b and the secondary target branch group 2c, and an output module 2f located at the output end of the collaborative fusion module 2d and the multi-concept intersection module 2e.

[0034] It should be noted that the backbone feature extraction module 2a uses the Swin Transformer as a shared backbone network, combined with the Feature Pyramid Network (FPN), to output multi-scale feature maps. The Swin Transformer-Tiny serves as the feature extraction backbone, and the FPN serves as the multi-scale feature fusion module. The main task branch uses Mask R-CNN as the basic instance segmentation framework. Therefore, the Swin Transformer, FPN, and Mask R-CNN together constitute the main task recognition network. The Swin Transformer-Tiny and FPN are shared backbone feature extraction modules 2a, while Mask R-CNN is the head structure that receives the output features from the shared backbone feature extraction module 2a and performs two-stage detection and instance segmentation. These are not two independent backbone networks.

[0035] Therefore, the concept-constrained multi-task instance segmentation network structure of the present invention adopts a single-stream input, shared encoding, primary and secondary target collaboration, and concept consistency screening overall architecture. It takes high-definition microscopic images as input, and takes the bounding boxes of microplastic particles, instance masks and polymer categories as the main outputs. At the same time, it outputs the prediction results of multiple concept attributes related to microplastic recognition, including but not limited to color, transparency, morphological parameters and texture features.

[0036] (1) Backbone feature extraction module 2a: The backbone feature extraction module 2a receives a single microscopic image or a sub-block of a microscopic image as input, and is used to extract multi-scale features for use by the main task branch and the secondary target branch group. In a preferred embodiment, the backbone feature extraction module uses the initial feature extraction block of Swin Transformer-Tiny and the Feature Pyramid Network (FPN).

[0037] Specifically, the input image first enters the initial feature extraction block of Swin Transformer-Tiny, which consists of Patch Partition and Linear Embedding. The input image is divided into 4×4 non-overlapping image patches and linearly mapped to initial token features. Subsequently, these initial token features enter a deep backbone network composed of the remaining main feature extraction blocks. This deep backbone network includes four Swin Transformer layers, with 2, 2, 6, and 2 Swin Transformer modules in each layer, respectively. Image patch merging layers are set between adjacent layers to progressively reduce feature map resolution and progressively expand the number of channels, thereby obtaining backbone features with hierarchical semantic representation. The layers are arranged as follows: image patch segmentation layer, linear embedding layer, first Swin Transformer layer, image patch merging layer, second Swin Transformer layer, image patch merging layer, third Swin Transformer layer, image patch merging layer, fourth Swin Transformer layer, Feature Pyramid Network (FPN), and multi-scale feature maps (P2, P3, P4, P5). The feature maps output from the four layers correspond to different spatial resolutions, representing microplastic particles of varying sizes and morphological complexities. These feature maps are then input into the Feature Pyramid Network (FPN) for multi-scale fusion. The FPN fuses the features from each layer through lateral connections, top-down upsampling, and output convolutional layers to generate multi-scale output feature maps. This is for use in subsequent branches. The initial feature extraction block includes an image patch segmentation layer and a linear embedding layer; the deep backbone network includes a first Swing Transformer layer, an image patch merging layer, a second Swing Transformer layer, an image patch merging layer, a third Swing Transformer layer, an image patch merging layer, and a fourth Swing Transformer layer.

[0038] In one embodiment, the feature maps output by the four Swin Transformer layers correspond to different spatial resolutions, used to characterize microplastic particles with different particle size ranges and morphological complexities. The features from the Swin Transformer layers are further input into a Feature Processing Network (FPN) for multi-scale feature fusion. The FPN includes lateral connection layers, a top-down upsampling path, and an output convolutional layer. The backbone features of each Swin Transformer layer are first channel-aligned by a 1×1 convolutional lateral connection layer, then fused step-by-step through a top-down upsampling path to construct a pyramid feature with unified semantic expression, and finally, a multi-scale output feature map is generated by a 3×3 convolutional output convolutional layer.

[0039] (2) Main task branch 2b: Main task branch 2b uses a segmentation framework to receive multi-scale feature maps and outputs the main task's output. In this embodiment, the main task's output includes at least: particle bounding boxes and particle instance masks (main region layers). ), polymer category score and main task confidence score.

[0040] In this embodiment, the main task branch 2b adopts the two-stage instance segmentation head structure of Mask R-CNN to complete the generation of candidate regions for microplastic particles, target localization, polymer category identification and instance mask prediction.

[0041] Main task branch 2b is set as follows: Step B1: Receive the multi-scale feature maps (P2, P3, P4, P5) output by the backbone feature extraction module 2a, and generate candidate regions on each scale feature map by the Region Proposal Network (RPN). Step B2: Use the RoIAlign layer of the main task branch 2b to extract fixed-size instance region features from the scale feature map corresponding to the candidate region, and feed them into the bounding box regression head, class prediction head and mask prediction head respectively. In other words, candidate regions are generated by the Region Proposal Network (RPN), and corresponding features are extracted by RoIAlign. These features are then fed into the bounding box regression head, the category prediction head, and the mask prediction head to obtain particle bounding boxes, polymer category scores, and instance masks.

[0042] Step B3: The bounding box regression head, as the localization branch, is used to output the coordinate correction of the candidate particle bounding box to obtain the particle bounding box; the category prediction head, as the category prediction branch, is used to output the prediction score of each polymer category as the polymer category score; the mask prediction head, as the segmentation branch, is used to output the pixel-level instance mask corresponding to the candidate particle as the particle instance mask.

[0043] The output of the main task serves as the basic input for subsequent collaborative fusion modules and multi-concept intersection modules. Therefore, the main task branch 2b in this scheme does not independently undertake the low-level feature extraction function. Its feature extraction and multi-scale fusion are completed by the backbone feature extraction module 2a, while the main task branch 2b is mainly responsible for performing two-stage instance segmentation and category prediction based on the features output by the backbone feature extraction module 2a.

[0044] (3) Secondary target branch group 2c: Secondary objective branch group 2c outputs prediction results for multiple concept attributes in parallel, serving as the output of the secondary task. Concept attributes include color, transparency, scale, shape, and texture attributes. The output of secondary objective branch group 2c serves as a crucial basis for subsequent collaborative fusion and multi-concept intersection processing.

[0045] Secondary target branch group 2c is set as follows: Step C1: Receive the multi-scale feature map output by the backbone feature extraction module 2a and extract instance region features using RoIAlign; Step C2: Extract local features from the instance region features through two consecutive 3×3 convolutional layers, with a ReLU activation layer following each convolutional layer to obtain shared intermediate features of the concept; Step C3: Input the intermediate features of the concept into multiple parallel concept prediction heads to output prediction results for multiple concept attributes.

[0046] In one embodiment, each concept prediction head outputs a concept classification score, rank, or regression value after global average pooling and a fully connected layer. For local concepts requiring spatial localization, a 1×1 convolutional layer-based concept prediction head can be appended after the intermediate features of the concept. In this case, the prediction results of the concept attributes include a concept response layer. ,in k is the ordinal number of the concept attribute, and n is the total number of concept attributes.

[0047] (4) Collaborative Integration Module 2d: The collaborative fusion module 2D processes the output of the main task based on the output of the secondary task to obtain the single-concept additional feature region. To avoid missed detections.

[0048] Among them, the collaborative fusion module 2d receives the particle instance mask (i.e., the main region layer) output by the main task branch 2b. The concept response layers output by the secondary target branch group and the secondary target branch group. Perform a one-to-one spatial comparison of single concepts, with each concept corresponding to a concept label and generating a corresponding single-concept overlapping region. Single-concept additional feature region and spatial overlap The collaborative fusion module ultimately outputs 2D concept labels and the additional feature regions of each concept label for each concept. Concept tags are used to mark additional feature regions of a single concept. This refers to the conceptual response area not covered by the main task, and serves as the input to output module 2f.

[0049] Among them, single concept overlapping areas Main area layer With concept response layer intersection Single-concept additional feature region for Spatial overlap , For intersection, For the union, To be relatively complementary, This represents the number of pixels in the region.

[0050] It should be noted that during the training phase, the Co-fusion module 2d does not participate in gradient calculation; it is only used during the inference phase. That is, during training, the output of the main task branch 2b is primarily used to calculate the loss function and does not enter the Co-fusion module 2d. During the inference phase, the inputs to the Co-fusion module 2d (assuming it is used) include: instance region features, polymer category score, main task confidence score, and particle instance mask.

[0051] Therefore, for candidate regions that meet the condition of "weak instance region features and mask response results in the main task branch and high consistency of concept attributes", the collaborative fusion module 2d can output single-concept additional feature regions. By marking conceptual response areas not covered by the main task, we can reinforce real microplastic particles that may be missed or have low confidence.

[0052] (5) Multi-concept intersection module 2e: The multi-concept intersection module 2e outputs the response layers of each concept based on the secondary target branch group. (i.e., the output of the secondary task), calculate the intersection of all concept layers. And utilize it on the main area layer Processing is performed to generate the minimum result. Suspected missed detection areas and the main region without feature support region The minimum value output by the multi-concept intersection module 2e. Suspected missed detection areas and the main region without feature support region As input to the result summary and output module 2f, it is used for the final result summary and statistics.

[0053] The intersection of all concept layers for: , Minimum result for: , Suspected missed detection areas for: , Main region without feature support region for: .

[0054] Therefore, for particles with high confidence in the main task but lacking support from multiple concepts or whose concept attributes are significantly inconsistent with the predicted polymer category, the multi-concept intersection module 2e performs screening.

[0055] (6) Output module 2f: Output module 2f receives the output results of the main task (particle bounding boxes and particle instance masks output by main task branch 2b (main region layer)). ), polymer category score and main task confidence score), output results of secondary tasks (concept response layer output by secondary objective branch group 2c). ), Single-concept additional feature region output by the collaborative fusion module 2D And the minimum value result output by the multi-concept intersection module 2e. Suspected missed detection areas and the main region without feature support region The final recognition result is output.

[0056] The final identification result includes at least the maximum value result. and minimum result Statistical indicators (including single-concept replenishment rate) Maximum expansion rate Minimum retention rate Suspected false negative rate and featureless support rate The final identification parameters include particle bounding boxes, particle instance masks, polymer category scores, main task confidence scores, and concept attribute prediction results.

[0057] Specifically, the particle bounding boxes, particle instance masks, polymer category scores, and main task confidence scores output by the main task branch 2b, and the prediction results of concept attributes output by the secondary target branch group 2c, are first aggregated and aligned using instance-level matching. This unifies and associates the prediction results of the particle bounding boxes, particle instance masks, polymer category scores, main task confidence scores, and corresponding concept attributes for the same particle. Subsequently, the maximum value result is obtained through processing. and minimum result .

[0058] In this embodiment, the maximum value result for: , in, Main area layer, For single-concept additional feature regions, This is a suspected missed detection area.

[0059] , , , , , in, For single-concept replenishment rate, This represents the maximum expansion rate. Minimum retention rate This represents the suspected false negative rate. The support rate is without characteristics. To prevent small constants from being divided by zero.

[0060] In one embodiment, the output module 2f can also summarize and encode the prediction results of multiple concept attributes to generate concept explanation information corresponding to each instance, which explains why the instance is determined to be a specific polymer category, such as the color level, transparency level, morphology parameters and texture attribute results corresponding to the instance.

[0061] During the training phase, the main task branch 2b, the secondary objective branch group 2c, the collaborative fusion module 2d, and the multi-concept intersection module 2e jointly participate in parameter optimization; during the inference phase, inputting a single microscopic image is sufficient to obtain the microplastic particle identification results that meet the screening criteria.

[0062] Step S3: Train the concept-constrained multi-task instance segmentation network; The multi-task joint loss function is obtained by weighted fusion of the main task loss, concept task loss, concept-main task consistency loss and false detection suppression loss, and the concept-constrained multi-task instance segmentation network is trained to achieve joint training.

[0063] like Figure 3 As shown, step S3 specifically includes: Step S301: Input the training samples into the concept-constrained multi-task instance segmentation network and perform multi-task forward propagation; The training samples are input into the concept-constrained multi-task instance segmentation network.

[0064] The main task outputs include: particle bounding boxes and particle instance masks (main region layer). ), polymer category score, and main task confidence score.

[0065] Secondary task output: Prediction results for each concept attribute (concept response layer) ).

[0066] The outputs of the logic operation module, namely the collaborative fusion module 2d and the multi-concept intersection module 2e, only perform region comparison and filtering operations. They are used in the inference stage and do not participate in gradient calculation, so they are not used to train the loss function.

[0067] Step S302: Calculate the main task loss; In this embodiment, based on the output of the main task branch 2b and the instance-level annotation information, the main task loss is... Specifically, the instance mask loss is calculated based on the output of main task branch 2b and the instance-level annotation information. Polymer classification loss Weighted fusion yields the main task loss. : , in, and To preset weights, For instance mask loss, For polymer classification loss.

[0068] Optionally, in other embodiments, the bounding box regression loss may be further calculated. and confidence score loss And merge it into the main task loss: , in, , , , To preset weights, For instance mask loss, For polymer classification loss, For bounding box regression loss, This represents the confidence score loss.

[0069] During the training phase, granular bounding boxes are used for bounding box regression loss. and polymer classification loss The calculation.

[0070] Step S303: Calculate the conceptual task loss; In this embodiment, based on the prediction results of the concept attributes and the labeled concept tags, the loss of each concept attribute (such as classification loss, regression loss or pixel-level segmentation loss) is calculated separately, and then weighted and fused according to preset weights to obtain the concept task loss.

[0071] Based on the output of secondary task branch 2c and the concept labels, the concept task loss is... for: , in, For the first Prediction results for each concept For the first A concept label, For the first Each concept loss weight, This refers to the corresponding loss function. Loss functions include classification loss, regression loss, or pixel-level segmentation loss.

[0072] Step S304: Obtain the total training loss function based on the main task loss and the concept task loss; Among them, the main task loss and concept task loss The data is fused according to preset fusion weights to obtain the total training loss function. ; Total training function for: , in, Loss to the main task For the loss of the conceptual task, and The weights are used to balance the contributions of the main task and the conceptual task during training.

[0073] Step S305: Continuously calculate the total loss function and update the parameters of the concept-constrained multi-task instance segmentation network until the training termination condition is met, at which point the trained concept-constrained multi-task instance segmentation network is obtained.

[0074] Among them, according to the total loss function Backpropagation and gradient updates are performed on the parameters of each module of the concept-constrained multi-task instance segmentation network until the training termination condition is met.

[0075] Step S4: Result Enhancement and False Detection Suppression in the Inference Stage: In the inference stage, the microscopic image to be identified is input into the trained concept-constrained multi-task instance segmentation network to obtain the final recognition result.

[0076] The final identification result includes at least the maximum value result. and minimum result Statistical indicators (including single-concept replenishment rate) Maximum expansion rate Minimum retention rate Suspected false negative rate and featureless support rate The final identification parameters include particle bounding boxes, particle instance masks, polymer category scores, main task confidence scores, and concept attribute prediction results.

[0077] The specific steps of step S4 are consistent with the working process of the conceptually constrained multi-task instance segmentation network described above. Step S4 specifically includes: Step S401: Forward inference of the model; The microscopic image to be identified is input into the trained concept-constrained multi-task instance segmentation network to obtain the bounding box of the particle, instance mask, polymer category score, main task confidence score, and prediction results of the corresponding concept attributes.

[0078] Step S402: Use the collaborative fusion module 2d to extract the main task response region and each concept response region for each candidate instance, perform spatial alignment and consistency calculation, generate a concept consistency guidance map, and perform weighted enhancement processing on the instance region features and mask response results of the main task output to obtain enhanced instance features and enhanced mask response. Step S403: Using the multi-concept intersection module 2e, perform filtering based on the multi-concept consistency score and the main task confidence score output by the main task branch 2b to obtain the particles that pass the filtering and send them to the output module.

[0079] Step S405: Output the final recognition result using output module 2f, which includes at least: the particle bounding box of the selected particles, the particle instance mask, the polymer category score, the main task confidence score, and the prediction results of the corresponding concept attributes.

[0080] 1. This invention introduces conceptual attributes such as color, transparency, shape, and texture as secondary tasks, in addition to the primary task of microplastic instance segmentation. This incorporates highly relevant conceptual attributes for microplastic identification into explicit supervision, enabling the model to be directionally constrained to learn intermediate semantic features related to microplastic identification. The discrimination criteria are interpretable, solving the problems of black-box feature extraction and uncontrollable focus in traditional models. Since these conceptual attributes can be defined, labeled, and selected by researchers, the model's learning direction is more controllable, and the results are more interpretable.

[0081] 2. This invention uses a collaborative fusion module to spatially align, calculate consistency, and weighted enhance the response regions shared by the main task and conceptual attributes. It enhances real microplastic particles with weak main task response but high conceptual consistency. While retaining the basic recognition capability of the main task, it can enhance the response of particles that conform to conceptual attributes but have insufficient main task response, thereby reducing missed detections.

[0082] 3. This invention uses a multi-concept intersection module to jointly screen based on multi-concept consistency scores and main task confidence, eliminating erroneous candidates with high main task scores but lacking concept support or whose concept attributes contradict the category, thereby achieving false detection suppression and misclassification removal. This solves the problems of easy confusion between similar polymers and easy misidentification due to background interference, thus reducing false detection and misclassification.

[0083] 4. This invention uses instance segmentation rather than simple object detection as the main task expression form, enabling the model to process microplastic particles with blurred boundaries, irregular shapes, complex local textures, and significant changes in transparency at the pixel level. It can accurately characterize the irregular contours, local textures, transparency distribution, and surface defects of microplastics, and is more suitable for the fine recognition needs of microplastics than traditional bounding box detection, thus improving the ability to recognize small particles and complex shapes.

[0084] 5. This invention uses the main task loss, concept task loss, consistency loss and false detection suppression loss to obtain the total loss in order to achieve joint optimization. This enables the shared backbone to learn instance localization, segmentation, category discrimination and concept attribute expression at the same time, and maintain stable recognition accuracy in microscopic image scenes with complex backgrounds, blurred boundaries, irregular shapes and large differences in transparency.

[0085] 6. This invention uses the Swing Transformer+FPN coding structure as a shared backbone feature extraction module, which balances multi-scale feature extraction efficiency and recognition accuracy. Concept types, network backbone and instance segmentation framework can be flexibly replaced, so that this invention can be deployed in offline scientific research and analysis scenarios, as well as in online detection equipment with microscopic imaging input, portable recognition terminals or laboratory automation platforms.

[0086] The backbone feature extraction module can use ResNet, ConvNeXt, Swin Transformer, VisionTransformer, CSPDarknet, or other network structures that can output multi-scale features, and is not limited to any specific network.

[0087] The main task instance segmentation module can adopt various segmentation frameworks, such as a two-stage instance segmentation framework, a single-stage instance segmentation framework, a query-based mask segmentation framework, or an improved structure thereof. As long as it can output instance-level masks and category results, it falls within the scope of this invention.

[0088] The types of concepts in the concept task module can be added or removed according to data conditions. In addition to color, transparency, shape and texture, they can also be expanded to reflectivity, edge sharpness, surface porosity, fragmentation and contamination level.

[0089] The steps of the collaborative fusion module can be implemented through the explicit intersection and union operations mentioned above, or through other set operations.

[0090] Concept labels can be directly labeled manually, or obtained through image rule calculation, statistical binning, expert rule generation, or semi-automatic labeling systems. As long as they can effectively supervise the concept task, they can be used as alternative implementation methods of this invention.

[0091] In summary, the microplastic species identification method of the present invention sets up multiple conceptual tasks in addition to the main task, and establishes a collaborative fusion module and a multi-concept intersection module. On the basis of utilizing the capabilities of the main task to complete microplastic particle instance segmentation and species identification, it further utilizes explicitly learned conceptual attributes of microplastics such as color, transparency, morphology, and texture to supervise the results of the main task. This supervision enhances the correct target response and filters out erroneous responses that do not conform to conceptual constraints. The output results of the main task branches of microplastic particle instance segmentation and polymer category identification are enhanced and filtered, thereby improving the accuracy of microplastic identification, reducing the false negative rate and false positive rate, and enhancing the interpretability and controllability of the model results.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. All simple and equivalent changes and modifications made in accordance with the claims and description of this application fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A method for identifying microplastic species based on concept-constrained multi-task instance segmentation, characterized in that, include: Step S1: Perform instance-level annotation and concept labeling on the microscopic images containing microplastic particles to obtain the input sample; Step S2: Construct a concept-constrained multi-task instance segmentation network with microplastic particle instance segmentation and polymer category identification as the main task and concept label prediction as the secondary task. The concept-constrained multi-task instance segmentation network processes the output of the main task based on the output of the secondary task to obtain additional feature regions for single concepts, avoiding missed detections. The minimum value result is generated by intersecting the output of the secondary task and the output of the main task, thus obtaining the final recognition result. Step S3: Train the concept-constrained multi-task instance segmentation network; Step S4: Input the microscopic image to be identified into the trained concept-constrained multi-task instance segmentation network to obtain the final recognition result.

2. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 1, characterized in that, The concept-constrained multi-task instance segmentation network as a whole includes a shared backbone feature extraction module, a main task branch and a secondary target branch group located at the output end of the backbone feature extraction module, a collaborative fusion module located at the output end of the main task branch and the secondary target branch group, a multi-concept intersection module located at the output end of the main task branch and the secondary target branch group, and an output module located at the output end of the collaborative fusion module and the multi-concept intersection module. The collaborative fusion module processes the output of the main task based on the output of the secondary task to obtain single-concept additional feature regions, thus avoiding missed detections. The multi-concept intersection module generates the minimum value result by intersecting the output results of the secondary task and the output results of the primary task.

3. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 2, characterized in that, The backbone feature extraction module receives a single microscopic image or a sub-block of a microscopic image as input, and is used to extract multi-scale features for use by the main task branch and the secondary target branch group. The main task branch employs a segmentation framework and outputs the main task's results, which include at least: particle bounding boxes, particle instance masks, polymer category scores, and main task confidence scores. The secondary task branch group outputs predictions for multiple concept attributes in parallel, serving as the secondary task's results. The concept attribute predictions include a concept response layer. ,in k is the ordinal number of the concept attribute, and n is the total number of concept attributes; The collaborative fusion module receives the granular instance mask output by the main task branch and the concept response layers output by the secondary target branch group, performs a one-to-one spatial comparison of single concepts, assigns a concept label to each concept, and generates corresponding single-concept overlapping regions. Single-concept additional feature region and spatial overlap The collaborative fusion module ultimately outputs concept labels and the additional feature regions of each concept label for each single concept. ; The multi-concept intersection module outputs each concept response layer based on the secondary target branch group. Calculate the intersection of all concept layers. And utilize it on the main area layer Processing is performed to generate the minimum result. Suspected missed detection areas and the main region without feature support region ; The output module receives the output results of the main task and the secondary task, and also receives the single-concept additional feature region output by the collaborative fusion module. And the minimum value result output by the multi-concept intersection module. Suspected missed detection areas and the main region without feature support region Output the final recognition result; The final identification result includes at least the maximum value result. and minimum result Statistical indicators and final identification parameters, including single-concept replenishment rate. Maximum expansion rate Minimum retention rate Suspected false negative rate and featureless support rate The final identification parameters include particle bounding boxes, particle instance masks, polymer category scores, main task confidence scores, and concept attribute prediction results.

4. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 3, characterized in that, The backbone feature extraction module uses the initial feature extraction block of Swin Transformer-Tiny and the Feature Pyramid Network (FPN); the main task branch uses the two-stage instance segmentation head structure of Mask R-CNN.

5. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 3, characterized in that, In the collaborative fusion module, The intersection of all concept layers for: , Minimum result for: , Suspected missed detection areas for: = , Main region without feature support region for: = , in, For concept response layers, where k is the ordinal number of the concept attribute, n is the total number of concept attributes; M is the coating of the main region.

6. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 5, characterized in that, Maximum value result for: ， in, Main area layer, For single-concept additional feature regions, Areas suspected of being missed during detection; ， ， ， ， ， in, For single-concept replenishment rate, This represents the maximum expansion rate. Minimum retention rate This represents the suspected false negative rate. The support rate is without characteristics. To prevent small constants from being divided by zero.

7. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 3, characterized in that, Step S3 specifically includes: Step S301: Input the training samples into the concept-constrained multi-task instance segmentation network and perform multi-task forward propagation; Step S302: Calculate the main task loss; Step S303: Calculate the conceptual task loss; Step S304: Obtain the total training loss function based on the main task loss and the concept task loss; Step S305: Continuously calculate the total loss function and update the parameters of the concept-constrained multi-task instance segmentation network until the training termination condition is met, at which point the trained concept-constrained multi-task instance segmentation network is obtained.

8. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 7, characterized in that, In step S302, the instance mask loss is calculated based on the output of the main task branch and the instance-level annotation information. Polymer classification loss Weighted fusion yields the main task loss. ; In step S303, the conceptual task loss for: ， in, For the first Prediction results for each concept For the first A concept label, For the first Each concept loss weight, This refers to the corresponding loss function; the loss function includes classification loss, regression loss, or pixel-level segmentation loss. In step S304, the main task loss is... and concept task loss The data is fused according to preset fusion weights to obtain the total training loss function. .

9. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 1, characterized in that, Step S1 specifically includes: Step S101: Acquire microscopic images containing microplastic particles and perform preprocessing on the original microscopic images; Step S102: Perform instance-level annotation on each microplastic particle in the microscopic image. The instance-level annotation information includes at least the instance mask and polymer category label. Step S103: Label the concepts; Step S104: Establish unified coding rules to encode different concept tags and obtain the coding results of concept tags.

10. The microplastic species identification method based on concept-constrained multi-task instance segmentation according to claim 1, characterized in that, The concept labels include multiple categories such as color category, transparency level, aspect ratio level, roundness level, texture roughness level, edge sharpness level, and surface defect level.

Images