Instance segmentation method of drug particle images based on context understanding

Abstract

The present invention discloses a method for instance segmentation of pharmaceutical particle images based on contextual understanding, comprising: S1. obtaining local and global contextual feature information of the particle image; wherein the particle image is divided into a normal particle image and a tiny particle image; S2. segmenting the normal particle image features and the tiny particle image features to obtain normal particle segmentation results and tiny particle segmentation results. The present invention can segment particle images quickly and reliably, effectively reducing the loss of detail information.

Description

Technical Field

[0001] The present invention relates to the field of particle image segmentation, and in particular to a method for instance segmentation of drug particle images based on context understanding. Background Art

[0002] In pharmaceutical research and manufacturing, the shape, size, distribution, and surface characteristics of drug particles have a significant impact on their properties and performance. Therefore, image segmentation of drug particles is of great significance for quantitatively and qualitatively describing the characteristics of drug particles.

[0003] Drug particles can have complex shapes and diverse appearances in images, including irregular shapes, fragments, overlaps, and deformations, making accurate particle segmentation difficult. Drug particle images can also be affected by low contrast and noise, resulting in blurred particle boundaries or noise interference, which also makes image segmentation difficult. Furthermore, drug particle sizes can vary greatly within an image, from small to large or from large to small, meaning that particles of different scales must be considered during image segmentation.

[0004] In summary, due to the above situation, the segmentation of drug particle images has become difficult. Therefore, a drug particle image instance segmentation method based on context understanding is needed to solve the above problems. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to overcome the defects in the prior art and provide a drug particle image instance segmentation method based on context understanding, which can quickly and reliably segment particle images and effectively reduce the loss of detail information.

[0006] The context-understanding-based drug particle image instance segmentation method of the present invention comprises the following steps:

[0007] S1. Obtaining local and global context feature information of the particle image; wherein the particle image is divided into a normal particle image and a tiny particle image;

[0008] S2. Segment the normal particle image features and the tiny particle image features to obtain normal particle segmentation results and tiny particle segmentation results.

[0009] Furthermore, local and global context feature information of the particle image is obtained, including:

[0010] Obtaining a feature map of a particle image;

[0011] Capturing feature patches in feature maps The local context within the local context is obtained to obtain the local context feature relationship The specific steps include:

[0012] S11. Feature blocks A single pixel within is considered as a graph node;

[0013] S12. Define the spatial and feature dependencies between graph nodes:

[0014]

[0015] in, For graph nodes dependencies, as well as Both expressed A b*b convolution operation with an output channel of C / 2 is used, and T is the transposed symbol;

[0016] S13. Feature representation of graph nodes and graph node dependencies in the interaction space through d-dimensional convolution U g and V g Interact and diffuse to aggregate the local context information around the graph nodes and obtain a graph structure that aggregates the local context information; where U g and V g Both are d-dimensional convolutions;

[0017] S14. Map the graph structure that aggregates local context information into feature relationships through convolution W

[0018] Where I is the identity matrix;

[0019] Based on the image classification model, global context dependency features are extracted from the feature patches, including:

[0020] A pyramid vision transformer is used to extract global context dependency features for feature map blocks in each network layer; wherein the pyramid vision transformer includes a block embedding layer and an encoder that encodes the feature map;

[0021] The block embedding layer divides the feature map into multiple feature map blocks and encodes the position information of the feature map blocks;

[0022] The encoder performs multi-scale semantic feature extraction on each feature map block to obtain the global dependency of the feature map.

[0023] Furthermore, the improved feature pyramid network is used to extract contextual features from tiny particle images, specifically including:

[0024] A tiny convolution branch is added to the feature pyramid network. The tiny convolution branch acts directly on the particle image and uses K L*L convolutions. Batch normalization and ReLU activation are performed after each convolution operation. Jump connections are used between every two convolutions for feature fusion.

[0025] Furthermore, it also includes: using an asymmetric pyramid network to enhance local and global context feature information, specifically including:

[0026] The feature map X i The convolution is then performed to reduce the dimension, and the dilated convolution operations of u*v and v*u are performed in the orthogonal dimensions with a dilation rate r to obtain the features of each layer.

[0027] Features of the jth layer in an asymmetric pyramid network It can be calculated by the following formula:

[0028]

[0029] Among them, f(X i ) represents the feature map X i Perform n*n convolution operation, μ r 、 They represent u*v and v*u dilated convolutions with dilation rate r, is the feature of the j-1th layer in the asymmetric pyramid network;

[0030] After obtaining features of multiple layers with different receptive fields, the multi-layer features are fused using n*n convolution to represent the multi-scale information of the feature pyramid network layer i in different receptive fields. The feature fusion method of the i-th network layer of the feature pyramid is as follows:

[0031]

[0032] Among them, C i ′ is the feature of the i-th network layer of the feature pyramid, and s is the number of layers of the asymmetric pyramid network.

[0033] Furthermore, the step S2 specifically includes:

[0034] Construct an instance segmentation head module to segment the normal particle image features and the tiny particle image features, and obtain the normal particle segmentation results and tiny particle segmentation results with different label colors;

[0035] The instance segmentation head module includes an object detection head, a semantic segmentation head and a fusion module;

[0036] The target detection head is used to realize bounding box positioning and target classification.

[0037] The semantic segmentation head is used to implement semantic segmentation of nanoparticles.

[0038] The fusion module is used to fuse the detected particle targets and semantic segmentation results.

[0039] Furthermore, the bounding box positioning specifically includes:

[0040] The bounding box is located by calculating the intersection-over-union ratio of the predicted bounding box and the true bounding box pixel by pixel. The center degree is used to suppress the predicted box whose distance from the true bounding box exceeds a set distance.

[0041] Determine the centrality according to the following formula * :

[0042]

[0043] Among them, l * 、r * , t * 、b * Represents the distance from the center point to the left, top, right, and bottom of the bounding box respectively.

[0044] Furthermore, the DR loss function is used to classify each pixel on the feature map, specifically including:

[0045] First, a confidence pair of positive object and negative background classes is generated for each bounding box, and then the confidence of positive and negative samples is sorted, and the sorting result is used to achieve target classification;

[0046] Among them, a classifier is obtained so that the loss function L DR (θ) minimum:

[0047]

[0048] Among them, θ represents a classifier, K represents the number of categories, represents the maximum confidence expectation of the negative sample, represents the minimum confidence expectation of the positive sample, γ represents the interval constant, l logistic represents the folding function, and its formula is: in, Represents the predicted result, and z represents the correct category.

[0049] Furthermore, the semantic segmentation of the nanoparticles specifically includes: constructing a loss function for semantic segmentation, and performing semantic segmentation using the loss function as a constraint;

[0050] The loss function L for semantic segmentation is determined according to the following formula: sem_seg :

[0051] Lsem_seg =αL small_mask +βL normal_mask ;

[0052] Among them, L small_mask is the cross entropy loss function of tiny particles, L normal_mask is the cross entropy loss function of normal particles, α and β are the weights of tiny particles and normal particles respectively, and α is greater than β.

[0053] Furthermore, the detected particle targets and semantic segmentation results are fused, including:

[0054] Use RoIPooler in Mask-RCNN to crop the corresponding region mask on mask B and resize it to a feature map of fixed size R×R. Then, use RoIAlign and Reshape to adjust the detection box P and its corresponding attention coefficient A to the same size as the feature map. Finally, multiply the detection box masks of different sizes and the corresponding attention coefficients to obtain instance segmentation masks with different label colors.

[0055] Among them, the mask B is obtained through semantic segmentation; the detection box P is obtained through bounding box positioning; and the attention coefficient A is obtained through the prediction of the target detection head.

[0056] The beneficial effects of the present invention are as follows: the present invention discloses a method for instance segmentation of drug particle images based on contextual understanding, which performs feature representation on nanoparticle images and uses instance segmentation methods to segment nanoparticles of different sizes one by one, and annotates each particle with a different label. This method can solve the problems of low contrast, blurring, and overlapping images caused by shooting conditions in grayscale nanoparticle images. To preserve and enhance the detailed information of tiny particles, the method does not extract information from the downsampled image, but directly extracts detailed feature information from the original image using multiple convolution operations, which can effectively reduce the loss of detailed information. The method adopts a multi-branch segmentation and weighted loss function strategy, which can better segment nanoparticles of different sizes in sparse or dense distributions, and reduce the difference in segmentation effect between tiny particles and normal particles. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0058] Figure 1 Schematic diagram of the overall process of the drug particle image instance segmentation method of the present invention;

[0059] Figure 2 Schematic diagram showing local and global context features of the present invention;

[0060] Figure 3Schematic diagram of the asymmetric pyramid network structure of the present invention;

[0061] Figure 4 Schematic diagram of the network structure of the segmentation head module of the present invention. DETAILED DESCRIPTION

[0062] The present invention is further described below with reference to the accompanying drawings, as shown in the drawings:

[0063] The context-understanding-based drug particle image instance segmentation method of the present invention comprises the following steps:

[0064] S1. Obtaining local and global contextual feature information of the particle image; wherein the particle image can be classified into a normal particle image and a small particle image according to the number of pixels occupied by the particle image; for example, if the number of pixels occupied is less than a set value, it is classified as a small particle image; otherwise, it is classified as a normal particle image;

[0065] S2. Segment the normal particle image features and the tiny particle image features to obtain normal particle segmentation results and tiny particle segmentation results.

[0066] The present invention uses a pyramid structure to represent the characteristics of nanoparticle images:

[0067] Graph convolution is combined with the Pyramid Visual Transformer (PVT) to extract local and global contextual information from particle images, and then combine the contextual relationships to better represent the characteristic information of particles of different sizes. Micro-convolution is used to extract feature information from the original image to enhance the details of small particles, and this detail information is combined with semantic information to represent more discriminative small particle features.

[0068] Using an instance-based segmentation head, nanoparticles are first divided into normal and tiny particles. Separate segmentation training is then performed for these two sizes. To further enhance the segmentation of difficult samples (tiny particles), the network is constrained to prioritize these difficult samples, giving them a greater weight in the loss function. Finally, the two segmentation results are fused to produce a color-coded nanoparticle segmentation result. The segmented particle images can then be color-coded using visualization software to obtain particles with different color labels.

[0069] The local-global contextual feature representation aims to extract local-global feature dependencies in an image using graph convolution and transformers to obtain discriminative feature information. Micro-convolutions are used to enhance the detailed information of tiny particles, achieving a small particle image feature representation that contains both semantic and detailed information. An asymmetric pyramid network increases the receptive field of particles of different sizes. The instance segmentation head uses different loss functions to constrain the network to segment nanoparticles of different sizes, particularly tiny particles.

[0070] The present invention can be used to automatically segment drug particle images imaged by a microscope, providing technical support for drug particle analysis in subsequent pharmaceutical research.

[0071] In this embodiment, in order to extract the particle image features under different qualities of microscope imaging as much as possible, the present invention proposes a local-global context feature learning module for representing and extracting the local context and global context information of feature maps at different scales.

[0072] As attached Figure 2 As shown in the figure, the module mainly consists of two parts, the local context module and the global context module. The former uses graph convolutional reasoning to aggregate local dependencies, and the latter uses pyramid visual transformer to learn global dependencies.

[0073] Obtain local and global context feature information of the particle image, including:

[0074] Obtaining a feature map of the particle image; wherein the feature map can be obtained by processing the original particle image using a convolutional neural network;

[0075] Capture feature patches X in the feature map using graph convolutional reasoning pi The local context within the local context is obtained to obtain the local context feature relationship The specific steps include:

[0076] S11. Feature blocks A single pixel within is considered as a graph node;

[0077] S12. Define the spatial and feature dependencies between graph nodes:

[0078]

[0079] in, For graph nodes dependencies, as well as Both expressed A 1*1 convolution operation with an output channel of C / 2 is used, and T is the transposed symbol;

[0080] S13. Feature representation of graph nodes and graph node dependencies in interaction space through 1D convolution U g and V g Interact and diffuse to aggregate the local context information around the graph nodes and obtain a graph structure that aggregates the local context information; where U g and V g Both are 1-dimensional convolution;

[0081] S14. The graph structure that aggregates local context information is mapped into feature relationships through a 1*1 convolution W with a channel number of C.

[0082] Where I is the identity matrix;

[0083] The module learns local context feature relationships Afterwards, the present invention uses a pyramid vision transformer to extract global context dependency features for the feature patches in each network layer.

[0084] Based on the image classification model, global context dependency features are extracted from the feature patches, including:

[0085] A pyramid vision transformer is used to extract global context dependency features for feature map blocks in each network layer; wherein the pyramid vision transformer includes a block embedding layer and an encoder that encodes the feature map;

[0086] The block embedding layer divides the feature map into multiple feature map blocks and encodes the position information of the feature map blocks;

[0087] The encoder performs multi-scale semantic feature extraction on each feature map block to obtain the global dependency of the feature map.

[0088] In this embodiment, real SEM microscope images contain many tiny particles. When extracting features from these images, detailed information decreases or even disappears as the deep learning network deepens. Therefore, to reduce the loss of tiny particle information, the present invention adds a small convolution branch to the traditional feature pyramid network (FPN) to enhance detailed information.

[0089] The tiny convolution branch is a subnetwork specifically designed to process subtle features and local details. It uses small convolution kernels to extract fine-grained feature information, thereby enhancing the capabilities of convolutional neural networks. This branch operates directly on the image, employing six 3x3 convolutions. Each convolution operation is followed by batch normalization and ReLU activation to address overfitting. Skip connections are used between each convolution for feature fusion. These tiny convolution features are fused with the bottom layer of the feature pyramid network to extract more detailed and semantic information, playing a crucial role in the segmentation of tiny particles.

[0090] In this embodiment, to further improve the segmentation accuracy of drug particles of different sizes, the present invention proposes an asymmetric pyramid network, which uses multiple dilated convolutions to enhance the feature information of each layer of the pyramid network from the perspective of receptive field to adapt to electron microscope scanning images of drug particles of different magnifications and sizes.

[0091] like Figure 3 As shown in Figure 1, the asymmetric pyramid is divided into 4 layers, each of which consists of four dilated convolutional layers with different dilation rates, i.e., dilation rates r = 2, 4, 6, and 8. It takes the features extracted by the local-global context module as input and fuses the features of each layer of the pyramid as output.

[0092] An asymmetric pyramid network is used to enhance local and global contextual feature information, specifically including:

[0093] The feature map X i The 1*1 convolution is sent to perform dimensionality reduction. In order to reduce the amount of calculation, 3*1 and 1*3 dilated convolution operations are performed in the orthogonal dimension with a dilation rate r to obtain the features of each layer.

[0094] Among them, the features extracted by the previous layer (such as the first layer) and the features after 1*1 convolution of the next layer (such as the second layer) are fused as the input of the next layer in the orthogonal dimension to better perform multi-scale feature representation.

[0095] Features of the jth layer in an asymmetric pyramid network It can be calculated by the following formula:

[0096]

[0097] Among them, f(X i ) represents the feature map X i Perform 1*1 convolution operation, μ r 、 They represent 3*1 and 1*3 dilated convolutions with dilation rate r, respectively. is the feature of the j-1th layer in the asymmetric pyramid network;

[0098] After obtaining features of multiple layers with different receptive fields, the multi-layer features are fused using n*n convolution to represent the multi-scale information of the feature pyramid network layer i in different receptive fields. The feature fusion method of the i-th network layer of the feature pyramid is as follows:

[0099]

[0100] Among them, C i ′ is the feature of the i-th network layer of the feature pyramid, s is the number of layers of the asymmetric pyramid network, and the value of s is 4.

[0101] In this embodiment, instance segmentation consists of target detection and semantic segmentation tasks. Therefore, in order to obtain the segmentation result of each nanoparticle, in step S2, an instance segmentation head module is constructed to segment the normal particle image features and the tiny particle image features to obtain the normal particle segmentation results and the tiny particle segmentation results with different label colors.

[0102] As attached Figure 4 As shown in Figure 2, the instance segmentation head module network consists of three parts: target detection head, semantic segmentation head and fusion module (Blender).

[0103] The target detection head is used to realize bounding box positioning and target classification.

[0104] The semantic segmentation head is used to implement semantic segmentation of nanoparticles.

[0105] The fusion module is used to fuse the detected particle targets and semantic segmentation results.

[0106] In this embodiment, the bounding box positioning specifically includes:

[0107] The bounding box is located by calculating the intersection-over-union ratio of the predicted bounding box and the true bounding box pixel by pixel. The center degree is used to suppress the predicted box whose distance from the true bounding box exceeds a set distance.

[0108] Determine the centrality according to the following formula * :

[0109]

[0110] Among them, l * 、r * , t * 、b * Represents the distance from the center point to the left, top, right, and bottom of the bounding box respectively.

[0111] Center suppression is used to suppress or weaken the center or obvious structure in the image in order to better focus on the edge or detail information.

[0112] In this embodiment, in order to solve the problem of particle sparseness in the electron microscope image of nanoparticles, that is, to make the network pay more attention to foreground pixels rather than background pixels, the DR loss function L is used. DR (θ) classifies each pixel on the feature map, that is, the loss function transforms the classification problem into a sorting problem.

[0113] Use the DR loss function to classify each pixel on the feature map, specifically including:

[0114] First, a confidence pair of positive object and negative background classes is generated for each bounding box, and then the confidence of positive and negative samples is sorted, and the sorting result is used to achieve target classification;

[0115] Among them, an ideal state is that the confidence of the positive sample is greater than the confidence of the negative sample. Therefore, the loss function expects the confidence of the positive sample to be as high as possible and the highest confidence of the negative sample to be as low as possible. Get a classifier so that the loss function L DR (θ) minimum:

[0116]

[0117] Among them, θ represents a classifier, K represents the number of categories, represents the maximum confidence expectation of the negative sample, represents the minimum confidence expectation of the positive sample, γ represents the interval constant, l logistic represents the folding function, and its formula is: in, Represents the predicted result, and z represents the correct category.

[0118] In this embodiment, pixel-level labeling of nanoparticles is achieved, that is, a mask is output. The traditional approach is to use a single-branch network combined with a cross-entropy loss function to compare each predicted pixel category with the labeled pixel category. When the two are extremely similar, the true pixel category is assigned to the predicted pixel.

[0119] To better segment nanoparticles of different sizes, the present invention divides the semantic segmentation head into two branches: one branch is used to segment tiny particles, and the other branch is used to segment other normal particles. The loss function of both branches uses the semantic segmentation cross-entropy loss function, but to make the network focus more on small particles, the present invention gives greater weight to the loss of tiny particles.

[0120] The semantic segmentation of the nanoparticles specifically includes: constructing a loss function for semantic segmentation, and performing semantic segmentation using the loss function as a constraint;

[0121] The loss function L for semantic segmentation is determined according to the following formula:sem_seg :

[0122] L sem_seg =αL small_mask +βL normal_mask ;

[0123] Among them, L small_mask is the cross entropy loss function of tiny particles, L normal_mask is the cross entropy loss function of normal particles, α and β are the weights of tiny particles and normal particles respectively, and α is greater than β.

[0124] In this example, to ultimately segment the nanoparticles and predict an instance label for each particle, the object detection and semantic segmentation results need to be fused. This fusion is achieved through a fusion module. This module takes the semantic segmentation result (mask B), the object detection result (detection box P), and the detection head's prediction result (attention coefficient A) as inputs and outputs a color mask for the instance label of each drug particle.

[0125] The detected particle targets and semantic segmentation results are fused, including:

[0126] Use RoIPooler in Mask-RCNN to crop the corresponding region mask on mask B and resize it to a feature map of fixed size R×R. Then, use RoIAlign and Reshape to adjust the detection box P and its corresponding attention coefficient A to the same size as the feature map. Finally, multiply the detection box masks of different sizes and the corresponding attention coefficients to obtain instance segmentation masks with different label colors.

[0127] Among them, the mask B is obtained through semantic segmentation; the detection box P is obtained through bounding box positioning; and the attention coefficient A is obtained through the prediction of the target detection head.

[0128] Mask-RCNN is a region-based convolutional neural network model used to detect and segment objects in images. RoIPooler is a key component in the Mask-RCNN model, used to extract features of regions of interest from feature maps. RoIAlign is a key component in the Mask-RCNN model, used to extract alignment features of regions of interest from feature maps. Reshape is a common tensor operation in deep learning; resize is a common image processing operation used to resize an image.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for instance segmentation of drug particle images based on context understanding, characterized by: The steps include: S1. Obtaining local and global context feature information of the particle image; wherein the particle image is divided into a normal particle image and a small particle image; obtaining local and global context feature information of the particle image, specifically including: Obtaining a feature map of a particle image; Capturing feature patches in feature maps The local context within the local context, get the local context feature relationship The specific steps include: S11. Feature blocks A single pixel within is considered as a graph node; S12. Define the spatial and feature dependencies between graph nodes: in, For graph nodes dependencies, as well as Both expressed A b*b convolution operation with an output channel of C / 2 is used, and T is the transposed symbol; S13. Feature representation of graph nodes and graph node dependencies in the interaction space through d-dimensional convolution U g and V g Interact and diffuse to aggregate the local context information around the graph nodes and obtain a graph structure that aggregates the local context information; where U g and V g Both are d-dimensional convolutions; S14. Map the graph structure that aggregates local context information into feature relationships through convolution W Where I is the identity matrix; Based on the image classification model, global context dependency features are extracted from the feature patches, including: Use pyramid vision transformer to extract global context dependency features for feature patches in each network layer; S2. Segment the normal particle image features and the tiny particle image features to obtain normal particle segmentation results and tiny particle segmentation results.

2. The method for instance segmentation of drug particle images based on context understanding according to claim 1, characterized in that: The improved feature pyramid network is used to extract contextual features from tiny particle images, specifically including: A tiny convolution branch is added to the feature pyramid network. The tiny convolution branch acts directly on the particle image and uses K L*L convolutions. Batch normalization and ReLU activation are performed after each convolution operation. Jump connections are used between every two convolutions for feature fusion.

3. The method for instance segmentation of drug particle images based on context understanding according to claim 1, characterized in that: Also includes: An asymmetric pyramid network is used to enhance local and global contextual feature information, specifically including: The feature map X i The convolution is then performed to reduce the dimension, and the dilated convolution operations of u*v and v*u are performed in the orthogonal dimensions with a dilation rate r to obtain the features of each layer. Features of the jth layer in an asymmetric pyramid network It can be calculated by the following formula: Among them, f(X i ) represents the feature map X i Perform n*n convolution operation, μ r 、 They represent u*v and v*u dilated convolutions with dilation rate r, is the feature of the j-1th layer in the asymmetric pyramid network; After obtaining features of multiple layers with different receptive fields, the multi-layer features are fused using n*n convolution to represent the multi-scale information of the feature pyramid network layer i in different receptive fields. The feature fusion method of the i-th network layer of the feature pyramid is as follows: Among them, C i ′ is the feature of the i-th network layer of the feature pyramid, and s is the number of layers of the asymmetric pyramid network.

4. The method for instance segmentation of drug particle images based on context understanding according to claim 1, characterized in that: The step S2 specifically includes: Construct an instance segmentation head module to segment the normal particle image features and the tiny particle image features, and obtain the normal particle segmentation results and tiny particle segmentation results with different label colors; The instance segmentation head module includes an object detection head, a semantic segmentation head and a fusion module; The target detection head is used to realize bounding box positioning and target classification. The semantic segmentation head is used to implement semantic segmentation of nanoparticles. The fusion module is used to fuse the detected particle targets and semantic segmentation results.

5. The method for instance segmentation of drug particle images based on context understanding according to claim 4, characterized in that: The bounding box positioning specifically includes: The bounding box is located by calculating the intersection-over-union ratio of the predicted bounding box and the true bounding box pixel by pixel. The center degree is used to suppress the predicted box whose distance from the true bounding box exceeds a set distance. Determine the centrality according to the following formula * : Among them, l * 、r * , t * 、b * Represents the distance from the center point to the left, top, right, and bottom of the bounding box respectively.

6. The method for instance segmentation of drug particle images based on context understanding according to claim 4, characterized in that: Use DR loss function L DR (θ) classifies each pixel on the feature map into target categories, including: First, a confidence pair of positive object and negative background classes is generated for each bounding box, and then the confidence of positive and negative samples is sorted, and the sorting result is used to achieve target classification; Among them, a classifier is obtained so that the loss function L DR (θ) minimum: Among them, θ represents a classifier, K represents the number of categories, represents the maximum confidence expectation of the negative sample, represents the minimum confidence expectation of the positive sample, γ represents the interval constant, l logistic represents the folding function, and its formula is: in, Represents the predicted result, and z represents the correct category.

7. The method for instance segmentation of drug particle images based on context understanding according to claim 4, characterized in that: The semantic segmentation of the nanoparticles specifically includes: constructing a loss function for semantic segmentation, and performing semantic segmentation using the loss function as a constraint; The loss function L for semantic segmentation is determined according to the following formula: sem_seg : L sem_seg =αL small_mask +βL normal_mask ; Among them, L small_mask is the cross entropy loss function of tiny particles, L normal_mask is the cross entropy loss function of normal particles, α and β are the weights of tiny particles and normal particles respectively, and α is greater than β.

8. The method for instance segmentation of drug particle images based on context understanding according to claim 4, characterized in that: The detected particle targets and semantic segmentation results are fused, including: Use RoIPooler in Mask-RCNN to crop the corresponding region mask on mask B and resize it to a feature map of fixed size R×R. Then, use RoIAlign and Reshape to adjust the detection box P and its corresponding attention coefficient A to the same size as the feature map. Finally, multiply the detection box masks of different sizes and the corresponding attention coefficients to obtain instance segmentation masks with different label colors. Among them, the mask B is obtained through semantic segmentation; the detection box P is obtained through bounding box positioning; and the attention coefficient A is obtained through the prediction of the target detection head.

Images