A method for instance segmentation of chemical fertilizer particles based on improved Mask R-CNN

Abstract

The application discloses a kind of based on improved Mask R-CNN's chemical fertilizer particle instance segmentation method.The method mainly includes four steps: collecting chemical fertilizer particle image and constructing image library;Image is enhanced, preprocessed and annotated dataset, then divides training set and test set;Improved Mask R-CNN network and trains the chemical fertilizer particle segmentation model based on the network;Based on the trained segmentation model, the chemical fertilizer particle image in test set is carried out instance segmentation, and instance segmentation result is obtained.Compared with the traditional segmentation network, the improved Mask R-CNN network of the application greatly improves the accuracy of chemical fertilizer particle segmentation.The method provided by the application can solve the shortcomings and deficiencies of the prior art for chemical fertilizer particle segmentation, and provide a basis for particle size statistical analysis of chemical fertilizer particle.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and in particular relates to a fertilizer particle instance segmentation method based on an improved Mask R-CNN. Background Technology

[0002] Fertilizers are considered the "food of grains" in agricultural production, playing a crucial role in increasing grain yields. Among the key characteristics of fertilizer products, particle size is a vital indicator of granulation quality. However, current domestic fertilizer granulation quality testing still relies on traditional methods of manual observation and random sampling. This method is time-consuming, labor-intensive, and highly subjective, offering poor real-time adaptability for granulation production adjustments. With the rapid development of deep learning and image processing technologies, remote, non-contact granulation quality inspection using computer vision has become possible.

[0003] To achieve intelligent detection of fertilizer granulation using computer vision, effective segmentation of fertilizer granule images is first required. However, fertilizer images present challenges such as dense granule distribution, wide particle size range, particle adhesion, and blurred contours. Traditional image segmentation algorithms fail to effectively separate fertilizer granules, resulting in undersegmentation of many small granules adhering to larger ones, while oversegmentation occurs due to occlusion between large granules. While the accuracy of segmentation using the original Mask R-CNN network model improves, the results are still unsatisfactory. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies for fertilizer particle segmentation and to provide a fertilizer particle instance segmentation method based on an improved Mask R-CNN. This method utilizes an improved Mask R-CNN network to segment fertilizer particle instances, effectively segmenting fertilizer particle images and laying the foundation for subsequent statistical analysis of particle size.

[0005] The technical solution of this invention to solve the above problems is: a fertilizer particle instance segmentation method based on improved Mask R-CNN, comprising the following steps:

[0006] S1. Collect images of fertilizer granules and construct an image library of fertilizer granules in fertilizer granulation production.

[0007] S2. Perform image augmentation and preprocessing on the collected fertilizer granule images, label the image dataset, and divide the image dataset into training set and test set;

[0008] S3. Analyze the features of fertilizer particle images, improve the Mask R-CNN network accordingly, and train a fertilizer particle segmentation model based on the improved Mask R-CNN network.

[0009] S4. Based on the trained instance segmentation model, fertilizer granule instance segmentation based on the improved Mask R-CNN is performed using the test set to obtain the instance segmentation results.

[0010] Preferably, the specific steps of step S1 are as follows:

[0011] S11. In the fertilizer production workshop, select a suitable image acquisition location and set up a fertilizer particle image acquisition device for the production process.

[0012] S12. Use camera client software to control the camera, take images of fertilizer granules at regular intervals, and create an image dataset.

[0013] S13. Use Zhang Zhengyou's camera calibration method to correct the image and obtain a more accurate image of fertilizer granules.

[0014] Furthermore, in step S13, the Zhang Zhengyou camera calibration method calculates the camera's intrinsic and extrinsic parameters by establishing the corresponding positional relationships between a series of three-dimensional points of the photographed object and two-dimensional points in the image. During calibration, a known calibration board is photographed from multiple angles to obtain a set of known three-dimensional points and corresponding two-dimensional points. The extraction and matching of the calibration board corner points at different angles are used to solve for the camera's intrinsic, extrinsic, and distortion parameters. These parameters can then be used for image correction, reconstruction, and other operations.

[0015] Preferably, the specific steps of step S2 are as follows:

[0016] S21. To improve the accuracy of the neural network model in segmenting fertilizer particles, it is essential to ensure both the quality and quantity of image data. Images containing blurred, motion-induced ghosting, or irrelevant fertilizer particles should be filtered and removed.

[0017] S22. Using image annotation software, label the outline of each particle in the image as a polygon, and save the annotation as a JSON file. Then use commands to construct the dataset format required for Mask R-CNN.

[0018] S23. To reduce the risk of overfitting and improve the generalization ability of the model, and because image annotation of fertilizer particles is a time-consuming and labor-intensive task, methods such as random cropping, random flipping, random rotation, and noise addition are used to augment the fertilizer particle image data.

[0019] S24. While expanding the original image, the corresponding masked image was expanded in the same way, and a total of 1000 granulation image segmentation datasets were finally obtained. These datasets were then randomly shuffled in an 8:2 ratio to form a training set and a test set.

[0020] Preferably, step S3, which improves the Mask R-CNN network based on the characteristics of fertilizer granule images, involves the following specific steps:

[0021] S31. When using Mask R-CNN for feature extraction, an edge feature extraction sub-network is connected in parallel. Then, the edge information output by the sub-network is added pixel by pixel to the original particle feature map, effectively integrating the information from both.

[0022] S32. Replace the ResNet backbone feature extraction network of the original Mask R-CNN with DenseNet-121 to better extract the features of fertilizer granules and improve the accuracy of granulation image segmentation.

[0023] S33. Overlapping fertilizer particles create darker holes in the image, leading to inaccurate fertilizer particle segmentation. A novel mask loss function is used to replace the original loss function of the Mask R-CNN network.

[0024] S34. Use the training set to train the instance segmentation model to obtain the trained instance segmentation model.

[0025] Furthermore, in step S31, the edge feature extraction sub-network uses the Laplacian edge detection algorithm to process the granulation image. The Laplacian algorithm uses a second-order differential operator to calculate the second derivative value of each pixel in the image; these second-order derivative values ​​can be regarded as edge information in the image. The mathematical definition of the Laplacian operator is:

[0026]

[0027] In digital image processing, the discrete form of the Laplacian operator can be expressed as:

[0028]

[0029] in The values ​​are on the horizontal axis. These are the values ​​for the vertical coordinate axes.

[0030] The main network extracts feature information from particles, while the edge feature extraction sub-network is specifically designed to extract feature information from particle edges. A fused feature map is obtained by superimposing the pixels of the two feature maps. This allows the network model to extract feature information from particles and edges more accurately, thereby optimizing particle segmentation.

[0031] Furthermore, in step S32, the backbone feature extraction network ResNet of the original Mask R-CNN is replaced with DenseNet. DenseNet uses a dense connection approach, unlike the skip connections of ResNet. Each Dense Block in the DenseNet network consists of multiple convolutional layers, and the output of each layer is directly connected to all subsequent layers, forming a densely connected path. This connection method can retain the information of all preceding layers, increase information flow and feature reusability, effectively reduce the number of network parameters, and improve the network's operating efficiency.

[0032] Furthermore, the new mask loss function is calculated in step S33 as follows:

[0033]

[0034] in:

[0035]

[0036] In the above formula Let be the category label of the i-th pixel in the candidate box. This represents the probability that the segmentation network predicts for the class of the i-th pixel, where n is the total number of pixels. and There are two hyperparameters, where Used to adjust the imbalance between positive and negative samples during training, by selecting appropriate... The value can force the mask segmentation network to focus on learning the features of edge pixels that are difficult to distinguish.

[0037] Furthermore, in step S34, the model is trained using the pre-defined training set and optimized using stochastic gradient descent (SGD) for 120 training rounds. To improve training efficiency and accuracy, the momentum in the experimental parameters is set to 0.9, the batch size to 4, and the initial learning rate to 0.001. During model training, the latest training parameters are saved to a designated file every other round, and finally, the optimal parameter model from all batches is selected.

[0038] Preferably, in step S4, the optimal parameter model trained in step S3 is used for testing on the test set. To verify the superiority of the improved Mask R-CNN algorithm for fertilizer granule image segmentation, multiple sets of comparative experiments were conducted. The traditional watershed segmentation algorithm, the U-net network (which excels in cell image segmentation in the biological field), and the original Mask R-CNN network were used to segment fertilizer granule images, respectively, verifying the superiority of the improved Mask R-CNN network in fertilizer granule image segmentation.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1) This invention utilizes an improved Mask R-CNN model for instance segmentation of fertilizer particles, effectively segmenting fertilizer particle images and laying the foundation for subsequent statistical analysis of particle size.

[0041] 2) In this invention, an edge feature extraction sub-network is connected in parallel during feature extraction. The backbone feature extraction network ResNet of the original Mask R-CNN is replaced with DenseNet. Furthermore, a new mask loss function replaces the loss function of the original Mask R-CNN network. Compared to traditional segmentation networks, the improved Mask R-CNN network significantly enhances the accuracy of fertilizer particle segmentation. Attached Figure Description

[0042] Figure 1 This is a flowchart of the overall segmentation algorithm of the present invention.

[0043] Figure 2 The present invention relates to a fertilizer granule image acquisition device.

[0044] Figure 3 This is a comparison image of the checkerboard pattern before and after distortion correction according to the present invention.

[0045] Figure 4 This is a schematic diagram of the segmentation mask for the labeled fertilizer granule image.

[0046] Figure 5 To augment the image in the segmentation dataset for the example of this invention.

[0047] Figure 6 The diagram shows the segmentation results of each segmentation algorithm of this invention. Detailed Implementation

[0048] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention can be combined accordingly, provided that there is no mutual conflict.

[0049] This invention provides a fertilizer particle instance segmentation method based on an improved Mask R-CNN, the specific process of which is as follows: Figure 1 This includes the following steps:

[0050] S1. Collect images of fertilizer granules and construct an image library of fertilizer granules in fertilizer granulation production.

[0051] In practical applications, the specific steps are as follows:

[0052] S11. In the fertilizer production workshop, select a suitable image acquisition location and set up an image acquisition device for fertilizer granules during the production process, such as... Figure 2 As shown. Includes: Label 1 represents the amino acid drum granulator; Label 2 represents the fertilizer outlet of the amino acid drum granulator; Label 3 represents the mounting bracket of the collection device; Label 4 represents a pair of symmetrically installed LED supplementary lights; Label 5 represents the camera; and Label 6 represents the fertilizer conveyor belt.

[0053] S12. Data collection: Use camera client software to control the camera, take images of fertilizer granules at regular intervals, and create an image dataset.

[0054] S13. The granulation image acquisition area established in this invention is approximately 250 mm × 250 mm, which is a large measurement field of view. Camera lens distortion is a common problem. This paper uses Zhang Zhengyou's camera calibration method to correct the image and obtain more accurate fertilizer granule images. The correction effect is verified by performing a correction test on a checkerboard image that completely fills the camera's field of view. The effects of the checkerboard image before and after distortion correction are shown below. Figure 3 As shown, Figure 3 Image a shows the checkerboard pattern before correction, and image b shows the checkerboard pattern after correction. It can be seen that the distortion has been effectively corrected.

[0055] S2. The collected fertilizer granule images are augmented and preprocessed, the dataset is labeled, and the image dataset is divided into training set and test set.

[0056] In practical applications, the specific steps are as follows:

[0057] S21. To improve the accuracy of the neural network model in segmenting fertilizer particles, it is necessary to ensure the quality and quantity of image data. Images containing blurry, motion ghosting, or irrelevant fertilizer particles in the fertilizer particle image library collected in step S1 are filtered out, retaining images with clear fertilizer particle features.

[0058] S22. Image data is labeled using the open-source software Labelme. This invention primarily achieves accurate segmentation of fertilizer granule images. For this purpose, polygons are fitted to the contours of the fertilizer granules using the labelme software, and the labeled data is saved as a JSON file. Then, the `labelme_json_to_dataset` command is used to construct the dataset format required for Mask R-CNN. Figure 4 The image shows a segmentation mask for an labeled fertilizer particle image, where (a) shows the location information of the fertilizer particles and (b) shows a visualization of the instance segmentation data.

[0059] S23. The amount of image data plays a crucial role in the effectiveness of model training. Training with a large amount of data can reduce the risk of overfitting and improve the model's generalization ability. However, image annotation of fertilizer granules is a time-consuming and labor-intensive task. The collected fertilizer granulation image set is highly similar to each other, and fertilizer granulation images still retain the semantic characteristics of normal granulation images even when horizontally or vertically flipped. When augmenting the granulation image segmentation data, vertical flipping, horizontal flipping, and scaling transformation are selected to effectively increase the number of samples while avoiding excessively high sample similarity. After augmentation, part of the dataset is shown below. Figure 5 As shown, Figure 5 (a) shows the original image and its mask image, (b) shows the image after scaling and its mask image, (c) shows the image after horizontal flipping and its mask image, and (d) shows the image after vertical flipping and its mask image.

[0060] S24. While augmenting the original images, the corresponding mask images are augmented in the same way, resulting in a total of 1000 granulation image segmentation datasets. These datasets are then randomly shuffled in an 8:2 ratio to form training and test sets.

[0061] S3. Analyze the features of fertilizer particle images, improve the Mask R-CNN network accordingly, and train a fertilizer particle segmentation model based on the improved Mask R-CNN network.

[0062] In practical applications, the specific steps are as follows:

[0063] S31. When using the Mask R-CNN feature extraction network to extract semantic features from granulation images, an edge feature extraction sub-network is run simultaneously to improve the response value of particle edges, thereby enhancing the network's segmentation ability. This sub-network uses the Laplacian edge detection algorithm to process fertilizer granule images to obtain accurate particle edge information. The edge information output by the sub-network is then added pixel-by-pixel to the original particle feature map, effectively integrating the information from both. This improves the response value of particle boundary regions in the feature map, resulting in better segmentation of fertilizer granules.

[0064] The Laplacian algorithm is a second-differential edge detection algorithm. Its basic principle is to represent edges in an image as extreme points of their second derivatives. The Laplacian algorithm uses a second-differential operator to calculate the second derivative value of each pixel in the image; these second derivative values ​​can be considered as edge information in the image. The mathematical definition of the Laplacian operator is:

[0065]

[0066] In digital image processing, the discrete form of the Laplacian operator can be expressed as:

[0067]

[0068] in The values ​​are on the horizontal axis. These are the values ​​for the vertical coordinate axes.

[0069] The main network extracts feature information from particles, while the edge feature extraction sub-network is specifically designed to extract feature information from particle edges. By superimposing the pixels of the two feature maps, a fused feature map is obtained. This network model can more accurately extract feature information from particles and edges, thereby optimizing particle segmentation. Figure 6 The image shown is a comparison chart of fertilizer granule characteristics.

[0070] S32. Because the ResNet backbone feature extraction network of the original Mask R-CNN did not perform well in segmenting cohesive particles, it was replaced with DenseNet to enhance the network's feature extraction capabilities. DenseNet uses a dense connection approach, unlike the skip connections in ResNet. Each Dense Block in the DenseNet network consists of multiple convolutional layers, and the output of each layer is directly connected to all subsequent layers, forming a densely connected path. This connection method can retain information from all preceding layers, increase information flow and feature reusability, effectively reduce the number of network parameters, and improve network efficiency.

[0071] S33. Fertilizer particles overlap, resulting in blurred particle outlines. Therefore, the mask loss function was modified to further improve the network's segmentation performance. The new mask loss function is shown in the following formula:

[0072]

[0073] in:

[0074]

[0075] In the above formula Let be the category label of the i-th pixel in the candidate box. This represents the probability that the segmentation network predicts for the class of the i-th pixel, where n is the total number of pixels. and There are two hyperparameters, where Used to adjust the imbalance between positive and negative samples during training, by selecting appropriate... This value forces the mask segmentation network to focus on learning the features of edge pixels that are difficult to distinguish. After multiple experimental comparisons, [the value is determined]. Set to 0.9, Set to 2. When training the network using stochastic gradient descent, minimize the mask loss function. The goal is to maximize the accuracy of single-pixel predictions while simultaneously improving the overall mask prediction results for each candidate box.

[0076] S34. The model was trained using a pre-defined training set and optimized using stochastic gradient descent (SGD) for 120 epochs. To improve training efficiency and accuracy, the momentum was set to 0.9, the batch size to 4, and the initial learning rate to 0.001. During training, the latest training parameters were saved to a specified file every epoch, and the optimal parameter model from all batches was selected at the end.

[0077] S4. Based on the trained instance segmentation model, fertilizer granule instance segmentation based on the improved Mask R-CNN is performed using the test set to obtain the instance segmentation results.

[0078] This invention proposes an improved Mask R-CNN algorithm for fertilizer granule image segmentation. To verify its superiority in granulation image segmentation, multiple sets of comparative experiments were conducted. For example... Figure 6 The image shows the segmentation results of various image segmentation algorithms on the granulation image. Among them, Figure 6 (a) in the image are the two original images from the segmentation test. Figure 6 Image (b) shows the segmentation results of the granulation image using the watershed algorithm. This algorithm tends to over-segment when there are small particles between large granules and large fertilizer granules in the image, causing large particles to be over-segmented into multiple pieces, which in turn leads to inaccurate subsequent fertilizer particle size statistics. Figure 6 (c) shows the results of granulation image segmentation using the U-net network. While U-net performs well in cell image segmentation in the biological field, it is not ideal for large, heavily aggregated fertilizer granules, where many granule boundaries remain unbroken, resulting in severe undersegmentation. Figure 6Figure (d) shows the results of the original Mask R-CNN segmentation algorithm. Although it achieves significant segmentation of the granulation image overall, some small fertilizer particles remain unidentified and unsegmented. Furthermore, the boundaries of severely adhered particles are not clearly segmented. Finally, Figure 6 Figure (e) shows the results of the improved Mask R-CNN segmentation algorithm. The improved Mask R-CNN network performs excellently in terms of the completeness of fertilizer particle segmentation, the number of small particles missed, and the effectiveness of boundary contour fitting. This indicates that the improved model proposed in this invention has better performance in granulation image segmentation compared to other segmentation networks, and can more accurately segment fertilizer particles, providing a more reliable foundation for subsequent analysis.

[0079] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for fertilizer particle instance segmentation based on improved Mask R-CNN, characterized in that, The method comprises the following steps: S1, collecting fertilizer particle images, and constructing a fertilizer particle image library in fertilizer granulation production; S2, performing image augmentation and preprocessing on the collected fertilizer particle images, labeling an image dataset, and then dividing the image dataset into a training set and a test set; S3, analyzing the characteristics of the fertilizer particle images, improving the Mask R-CNN network in a targeted manner, obtaining an improved Mask R-CNN network, and training a fertilizer particle segmentation model based on the improved Mask R-CNN network; S4, using the test set to perform instance segmentation of the fertilizer particles based on the improved Mask R-CNN network, and obtaining the instance segmentation result; In step S3, the specific steps of analyzing the characteristics of the fertilizer particle images and improving the Mask R-CNN network in a targeted manner are as follows: S31, when using Mask R-CNN to extract features from the fertilizer particle images, a parallel edge feature extraction subnetwork is connected; then the edge information output by the subnetwork is added to the original particle feature map pixel by pixel, effectively integrating the information of the two; S32, replacing the original Mask R-CNN backbone feature extraction network ResNet with DenseNet; S33, replacing the original Mask R-CNN network loss function with a new mask loss function; the calculation method of the new mask loss function is as follows: wherein: , is the class label of the i-th pixel in the bounding box, is the class prediction probability of the i-th pixel by the segmentation network, and n is the total number of pixels; and are two hyperparameters, wherein is used to adjust the imbalance of positive and negative samples in the training process, by selecting the value, to force the mask segmentation network to focus on learning the features of the difficult-to-distinguish edge pixels; S34, using the training set to train the instance segmentation model to obtain a trained instance segmentation model.

2. The improved Mask R-CNN-based granule instance segmentation method for chemical fertilizers according to claim 1, characterized in that, The specific steps of step S1 are as follows: S11, in the fertilizer production workshop, select the image acquisition position, and build a fertilizer particle image acquisition device in the production process; the fertilizer particle image acquisition device comprises a camera for shooting fertilizer particle images; S12, use the camera client software to control the camera, shoot the fertilizer particle images regularly, and make an image dataset; S13, use Zhang Zhengyou camera calibration method to correct the image, and obtain more accurate fertilizer particle images.

3. The improved Mask R-CNN-based granule instance segmentation method for chemical fertilizers according to claim 2, characterized in that, In step S13, the Zhang Zhengyou camera calibration method calculates the camera internal and external parameters by establishing the corresponding position relationship between a series of three-dimensional points of the shooting object and two-dimensional points in the image; In the calibration, the known calibration board is shot at multiple angles to obtain a group of known three-dimensional points and corresponding two-dimensional points; the extraction and matching of the calibration board corner points under different angles are used to solve the internal and external parameters and distortion parameters of the camera; the parameters are used for image correction and reconstruction operation.

4. The improved Mask R-CNN-based granule instance segmentation method for chemical fertilizers according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21, first, the images of the fertilizer particle images that are blurred, motion ghosting, and irrelevant fertilizer particles are screened out, and the images with clear fertilizer particle characteristics are retained; S22, using image labeling software, labeling the contour of each particle in the image, saving it as a json type file after labeling, and using a command line command to construct the dataset format required by Mask R-CNN; S23, augment the fertilizer particle image data; S24, while expanding the original image, the corresponding mask annotation image is also expanded, and the image data set is obtained and randomly divided into training set and test set according to a certain proportion.

5. The improved Mask R-CNN based granular instance segmentation method for chemical fertilizer according to claim 4, characterized in that, In step S23, the method for expanding the chemical fertilizer particle image data includes at least one of random cropping, random flipping, random rotation, and noise addition.

6. The improved Mask R-CNN based granular instance segmentation method for chemical fertilizer according to claim 4, characterized in that, In step S24, the number of images in the training set and the test set is 8:

2.

7. The improved Mask R-CNN-based granule instance segmentation method for chemical fertilizers according to claim 1, characterized in that, In step S31, the edge feature extraction sub-network adopts a Laplacian edge detection algorithm to process the granulation image; the Laplacian algorithm uses a second-order differential operator to calculate the second-order derivative value of each pixel in the image, and the second-order derivative value is regarded as the edge information in the image; the mathematical definition of the Laplacian operator is: In digital image processing, the discrete form of the Laplacian operator is represented as: wherein is the value of the horizontal coordinate axis, is the value of the vertical coordinate axis.

8. The improved Mask R-CNN-based granule instance segmentation method for chemical fertilizers according to claim 1, characterized in that, In step S4, the granular image of the chemical fertilizer is segmented by using a watershed segmentation algorithm, a U-net network excellent in cell image segmentation in the biological field, and an original Mask R-CNN network, and the superiority of the improved Mask R-CNN network in the granular image segmentation of the chemical fertilizer is verified.

Images