A method and system for abrasive chain segmentation based on an improved Mask-RCNN network

Abstract

The present invention discloses a method and system for abrasive chain segmentation based on an improved Mask-RCNN network. The contour information of abrasives is marked by the reflected light and transmitted light images of the abrasive chain collected by an online ferrograph, and a segmentation dataset is constructed. A network with multi-resolution parallel fusion is selected as the feature extraction layer. The IOU loss based on confidence is defined as the loss function of the abrasive bounding box regression branch. The loss functions of the abrasive classification branch, the bounding box regression branch, and the Mask branch are weighted and fused as the optimization objective of the abrasive chain segmentation model. The image to be segmented is input into the model, and the abrasive feature information is extracted according to the prediction results. The present invention is based on the improvement of the MaskR-CNN network, which solves the problem that the current abrasive chain segmentation technology cannot intelligently segment abrasive chain images.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine wear state monitoring, and particularly relates to a method and system for abrasive chain segmentation based on an improved Mask-RCNN network. Background Art

[0002] During the operation of mechanical equipment, the contact surface wear caused by the relative movement between friction pairs is inevitable. Wear is the basic form of component failure and the main cause of equipment faults. As the direct product of the wear process, abrasive particles carry a large amount of wear information, providing an important basis for wear mechanism analysis and wear state detection. Therefore, the abrasive particle analysis technology has become an important technical means in the field of mechanical equipment wear state detection.

[0003] A single abrasive particle can reflect the local wear condition of the friction pair at a certain moment, while the common characteristics of the abrasive particle group can reflect the current operating state of the equipment. Therefore, single abrasive particle analysis is the cornerstone for deducing the wear mechanism and equipment health state of the equipment. The ferrography technology uses magnetic force to adsorb abrasive particles, which inevitably results in the abrasive particles collected being distributed in a chain shape, increasing the difficulty of extracting and identifying the characteristics of single abrasive particles. For this reason, relevant scholars have explored the problem of "how to segment the abrasive particle chain and extract single abrasive particle information". Wu Hongkun et al. used the gray scale of the abrasive particle surface as prior information, established internal and external markers, and combined with the watershed algorithm to achieve single abrasive particle extraction. Wang Jingqiu et al. fused the watershed algorithm with the gray clustering algorithm (CMWGC), and through the correlation analysis of the color information and relative positions of each region after the initial segmentation by the watershed algorithm, and using the aspect ratio to discriminate and correct the segmentation results, achieved the rapid and accurate segmentation of abrasive particles. Chen Quansong proposed an abrasive particle chain segmentation method for ferrography images based on the nearest neighbor method. By using the nearest neighbor method to match the ferrography images at different times to obtain the deposition change sequence of the abrasive particle chain, and using the difference between two adjacent abrasive particle chains in the deposition change sequence to extract the single abrasive particle region. However, the characteristics of complex shapes and different sizes of abrasive particles result in the need to continuously optimize the model parameters of the above algorithms to achieve an ideal segmentation effect, which cannot meet the requirements of automatic detection.

[0004] In recent years, with the development of deep learning, the intelligent segmentation algorithm based on Mask R-CNN has been widely applied to the field of target detection due to its excellent segmentation ability. Mask R-CNN adds a Mask branch on the basis of Faster-RCNN, which can classify the pixels of the target while completing the target classification and bounding box tasks, that is, determine the contour of the target. However, due to the particularity of abrasive particles, the Mask R-CNN network has the following problems when segmenting abrasive particle chains:

[0005] 1) The Mask R-CNN network discriminates the consistency between the real region and the predicted position through the bounding box regression loss function. However, the shape, size, and distribution of abrasive particles are random, making it difficult to accurately locate the real position of abrasive particles.

[0006] 2) The Mask R-CNN network follows the normalization principle and needs to reduce the dimension of the candidate regions. Due to the different sizes of the abrasive particle feature maps, there are unpredictable pixel losses in the candidate regions during the dimension reduction process, resulting in the misalignment between the abrasive particle mask position and the real position.

[0007] In summary, the ferrography technology analyzes the abrasive particle information to characterize the wear evolution process, providing an effective means for the analysis of the wear mechanism of mechanical equipment and the health monitoring of equipment. However, the abrasive particles in the ferrography image are arranged in a chain, and the parameter optimization of the existing segmentation algorithms is complex and the segmentation accuracy is poor, restricting the effectiveness of the ferrography analysis technology. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide an abrasive particle chain segmentation method and system based on an improved Mask-RCNN network in view of the above deficiencies in the prior art. The original Mask R-CNN is improved to solve the technical problem that the segmentation mark (mask) cannot be completely aligned with the target region when Mask R-CNN is applied to abrasive particle chain segmentation, and accurate segmentation of the abrasive particle chain image is achieved.

[0009] The present invention adopts the following technical solutions:

[0010] An abrasive particle chain segmentation method based on an improved Mask-RCNN network, comprising the following steps:

[0011] S1. Extract the overall contour of the abrasive particle chain using the transmitted light image of the abrasive particle chain, manually segment adjacent abrasive particles according to the reflected light image of the abrasive particle chain, superimpose to obtain the single abrasive particle contour information in the abrasive particle chain image, and construct an abrasive particle chain segmentation data set according to the single abrasive particle contour information;

[0012] S2. Construct a multi-task parallel deep learning model based on the Mask R-CNN framework;

[0013] S3. Define the loss functions of each task branch of the multi-task branch layer, and weight and fuse the loss functions as the loss function L of the improved Mask-RCNN network loss ;

[0014] S4. Take the loss function L obtained in step S3 loss as the optimization target, use the abrasive particle chain segmentation data set constructed in step S1 to train the multi-task parallel deep learning model constructed in step S2, and then input the abrasive particle chain image to be segmented to obtain an abrasive particle mask image, realizing accurate segmentation of the abrasive particle chain image.

[0015] Specifically, step S1 is specifically as follows:

[0016] Collect the reflected light image and transmitted light image of the abrasive chain, extract the contour of the abrasive chain in the transmitted light image using the Sobel operator, combine the dividing lines of adjacent abrasives manually marked based on the reflected light image, and construct a complete segmentation label of the abrasive chain; repeat the above operations to establish an abrasive chain segmentation database, and divide it into a training set, a validation set, and a test set according to 8:1:1; use the training set as the input, verify the accuracy of the deep learning model through the validation set, and test the effect of the deep learning model through the test set.

[0017] Specifically, in step S2, the deep learning model includes:

[0018] A feature extraction layer, adopting the HR-NetV2p+FPN backbone network as the feature extractor for the abrasive chain image;

[0019] A region proposal layer, which is a lightweight convolutional neural network for finding the abrasive feature region;

[0020] A pooling layer, adopting the strategy of augmented pooling to unify the size of the ROI region;

[0021] A multi-task branch layer, including an abrasive mask branch, an abrasive classification branch, and an abrasive bounding box branch. The abrasive mask branch is a convolutional neural network, and both the abrasive classification branch and the abrasive bounding box branch are fully convolutional networks.

[0022] Furthermore, the upsampling of the feature extraction layer uses a 3×3 convolutional kernel with a stride of 1, and the downsampling of the feature extraction layer uses a 3×3 convolutional kernel with a stride of 2.

[0023] Specifically, in step S3, the loss function L loss is specifically as follows:

[0024] L loss = 0.2L cls + 0.4L box + 0.4L mask

[0025] where L cls represents the loss function of the abrasive classification branch, L box is the loss function of the bounding box regression branch, and L mask represents the loss function of the mask branch.

[0026] Furthermore, the loss function L box of the bounding box regression branch is:

[0027] L box = 1 - IoU

[0028]

[0029] Among them, A represents the predicted bounding box, B represents the ground truth bounding box, and IoU reflects the overlapping part between the predicted bounding box and the ground truth bounding box.

[0030] Specifically, step S4 is specifically as follows:

[0031] S401. Input the abrasive chain segmentation dataset into the deep learning model based on the Mask-RCNN network architecture;

[0032] S402. Set the learning rate, the exponential decay rate of the first moment, and the exponential decay rate of the second moment of the ADAM optimizer;

[0033] S403. Every n iterations, test the effect of abrasive chain segmentation once and save the parameters of the deep learning model;

[0034] S404. Screen out the model parameters with the best performance as the final result, input the abrasive chain image to be segmented into the trained deep learning model to obtain the abrasive mask image, and locate the real position of the abrasive according to the abrasive mask information, so as to realize the segmentation of single abrasives from the abrasive chain image.

[0035] Furthermore, in step S402, the learning rate is 0.001, the exponential decay rate of the first moment is 0.9, and the exponential decay rate of the second moment is 0.999.

[0036] Furthermore, in step S403, n = 2000.

[0037] In a second aspect, an abrasive chain segmentation system based on an improved Mask-RCNN network according to an embodiment of the present invention includes:

[0038] A data module, configured to extract the overall contour of the abrasive chain by using the transmitted light image of the abrasive chain, manually segment adjacent abrasives according to the reflected light image of the abrasive chain, obtain the single abrasive contour information in the abrasive chain image through superposition, and construct an abrasive chain segmentation dataset according to the single abrasive contour information;

[0039] An improvement module, configured to construct a multi-task parallel deep learning model based on the Mask R-CNN framework;

[0040] A loss module, configured to define the loss functions of each task branch of the multi-task branch layer, and fuse the loss functions with weights as the loss function L of the improved Mask-RCNN network loss ;

[0041] A segmentation module, configured to use the loss function L obtained by the loss module lossTaking the optimization objective, a multi-task parallel deep learning model constructed by the optimization and improvement module is optimized using the abrasive chain segmentation dataset constructed by the data module. Then, the abrasive chain image to be segmented is input to obtain the abrasive mask image, realizing the accurate segmentation of the abrasive chain image.

[0042] Compared with the prior art, the present invention has at least the following beneficial effects:

[0043] A method for segmenting abrasive chains based on an improved Mask-RCNN network according to the present invention marks the abrasive contour information through the reflected light and transmitted light images of the abrasive chains collected by an online ferrograph, constructs a segmentation dataset; selects a network with multi-resolution parallel fusion as the feature extraction layer; defines the IOU loss based on confidence as the loss function for the abrasive bounding box regression branch; weights and fuses the loss functions of the abrasive classification branch, the bounding box regression branch, and the Mask branch as the optimization objective of the abrasive chain segmentation model; inputs the image to be segmented into the model, and extracts the abrasive feature information according to the prediction result. The present invention is improved based on the Mask R-CNN network, solving the problem that the current abrasive chain segmentation technology cannot intelligently segment abrasive chain images.

[0044] Further, in step S1, the transmitted light image and the reflected light image of the abrasive chain are collected by online ferrography; the abrasive chain contour information in the transmitted light image of the abrasive chain is stronger, so the Sobel operator can be used to extract the abrasive chain contour information; the reflected light image of the abrasive chain has stronger abrasive surface texture information, which can be used as the basis for distinguishing the boundary between adjacent abrasives; therefore, the complete abrasive chain segmentation label can be constructed by combining the reflected light image and the transmitted light image of the abrasive chain; repeat the above operation to obtain the abrasive chain segmentation dataset; and divide the training set, validation set, and test set according to 8:1:1.

[0045] Further, in step S2, a multi-task parallel deep learning model based on the Mask R-CNN framework ensures that the feature image always has a high resolution during the convolution operation by fusing information from abrasive chain feature images at different resolutions, avoiding the loss of picture detail information as the convolution operation progresses; since the ROI region is obtained by bounding box regression and its coordinate values are floating-point numbers, if direct pooling operation is performed, the computer will quantize the coordinates to integers, and at this time, the bounding box will deviate from the bounding box obtained by regression, resulting in inaccurate prediction of the abrasive position. Therefore, before performing the pooling operation on the ROI region, the length and width of the bounding boxes of the candidate regions (ROIs) of different-scale RPNs need to be expanded to the smallest multiple of the pooling scale simultaneously, avoiding the quantization process and improving the accuracy of abrasive position prediction.

[0046] Furthermore, for flat sampling, a 3×3 convolutional kernel with a step size of 1 is used, which can retain the detailed information of the abrasive chain image to a large extent. For downsampling, a 3×3 convolutional kernel with a step size of 2 is used, which can reduce the computational amount of data and save the training time of the deep learning model.

[0047] Furthermore, since the classification objects in the abrasive chain segmentation task are only abrasives and the picture background, and they are easy to distinguish, while the bounding box and mask information of the abrasives are the main tasks, and the accuracy directly affects the effect of abrasive chain segmentation, the weight ratio is assigned as L cls : L box : L mask = 2:4:4.

[0048] Furthermore, the confidence IoU is the probability value of the Mask-RCNN network evaluating that the feature region belongs to the target region, which reflects the overlapping part between the predicted box and the ground truth box. At the same time, the confidence is obtained by jointly solving the center coordinates, length, and width of the predicted box and the ground truth box. While calculating the error between the predicted value and the true value, the correlation between the values is retained, which is consistent with the actual situation and has scale invariance.

[0049] Furthermore, the ADAM optimizer combines the Momentum optimizer and the RMSprop optimizer, and uses both the first-order momentum and the second-order momentum. The first-order momentum is determined not only by the gradient direction of the current point but also by the gradients accumulated previously, which increases the stability of the optimizer, speeds up the convergence rate, and has a certain ability to get rid of local optima. The second-order momentum is the sum of the squares of all gradient values, and by means of exponential weighting, the focus is concentrated on a certain period in the past, avoiding the problem that the second-order momentum continues to accumulate and leads to the premature end of the training process.

[0050] Furthermore, through training and parameter tuning, the optimal learning rate of the deep learning model parameters is determined to be 0.001, the exponential decay rate of the first moment is 0.9, and the exponential decay rate of the second moment is 0.999.

[0051] Furthermore, every 2000 iterations, the effect of abrasive chain segmentation is tested to prevent the underfitting caused by too few training times and the poor chain segmentation effect caused by overfitting due to too many training times.

[0052] It can be understood that the beneficial effects of the second aspect above can refer to the relevant descriptions in the first aspect above, and will not be elaborated here.

[0053] In summary, the present invention realizes the automatic segmentation of abrasive chain images, and at the same time solves the problems of inaccurate prediction of the true positions of abrasives and the misalignment between the mask positions and the true positions of abrasives when Mask-RCNN is directly applied to the abrasive chain segmentation task, improves the accuracy of abrasive chain segmentation, and provides technical support for the application of ferrography analysis technology.

[0054] Next, through the accompanying drawings and embodiments, the technical solutions of the present invention will be further described in detail. Brief Description of the Drawings

[0055] Figure 1 It is a flowchart of the abrasive chain segmentation method based on Mask-RCNN in the embodiment of the present invention;

[0056] Figure 2 It is a schematic diagram of the specific structure of the feature extraction layer of the Mask-RCNN network;

[0057] Figure 3 It is a schematic diagram of the network structure of the improved Mask-RCNN image segmentation algorithm of the present invention;

[0058] Figure 4 It is a network structure diagram of the improved Mask-RCNN image segmentation algorithm in the embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of an abrasive chain image sample in the embodiment of the present invention, where (a) is the abrasive chain image and (b) is the abrasive chain label;

[0060] Figure 6 It is a schematic diagram of the IOU calculation principle;

[0061] Figure 7 It is a diagram of the abrasive chain segmentation result in the present invention, where (a) is the abrasive chain segmentation result and (b) is the abrasive chain mask result. Detailed Embodiments

[0062] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0063] In the description of the present invention, it should be understood that the terms "include" and "comprise" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0064] It should also be understood that the terms used in the specification of the present invention are merely for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0065] It should be further understood that the term "and / or" used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this text generally represents an "or" relationship between the contextually related objects.

[0066] It should be understood that although terms such as first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0067] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0068] Structural schematic diagrams according to the disclosed embodiments of the present invention are shown in the drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of the various regions and layers shown in the figures and their relative sizes and positional relationships are merely exemplary, and in practice, there may be deviations due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0069] The present invention provides a method for segmenting abrasive chains based on an improved Mask-RCNN network, which uses reflected light images and transmitted light images to label the abrasive chain segmentation dataset; constructs a feature extraction network that fuses multi-resolution information by weight based on the HR-NetV2p network; establishes an abrasive chain segmentation model based on the Mask-RCNN framework, including an abrasive classification branch, a bounding box regression branch, and a Mask branch; and defines a loss function based on weighted fusion of losses of multi-task parallel branches to achieve precise segmentation of abrasive chain images, providing a guarantee for single abrasive feature extraction.

[0070] Please refer to Figure 1 , a method for segmenting abrasive chains based on an improved Mask-RCNN network according to the present invention, comprising the following steps:

[0071] S1. Establish an abrasive chain segmentation dataset;

[0072] Extract the overall contour of the abrasive chain using the transmitted light image of the abrasive chain, and manually segment adjacent abrasives according to the reflected light image of the abrasive chain. Superimpose the two to obtain the single abrasive contour information in the abrasive chain image, and construct an abrasive chain segmentation dataset;

[0073] S101. Collect the reflected light image and transmitted light image of the abrasive chain through an on-line ferrograph;

[0074] S102. Perform edge extraction on the transmitted light image using the Sobel operator to obtain the boundary contour of the abrasive chain;

[0075] S103. According to the different texture information on the surfaces of different abrasives in the reflected light image, manually mark the boundary line between adjacent abrasives, and add it to the result of S2 to obtain a complete abrasive chain segmentation label (in json format);

[0076] S104. Eliminate sample labels with poor quality and high similarity, convert the qualified samples into the standard coco dataset format, where the ratio of the training set, test set, and validation set is 8:1:1, and save all qualified samples in the abrasive chain segmentation database.

[0077] The reflected light image of the abrasive chain and the marked result of the abrasive contour are as Figure 5 shown, Figure 5 (a) is the reflected light image of the abrasive chain; Figure 5 (b) is the reflected light image of the abrasive chain with the marked abrasive contour, and the marked information of the abrasive contour is saved in the form of coordinate points.

[0078] S2. Construct a multi-task parallel deep learning model based on the Mask R-CNN framework;

[0079] The deep learning model includes:

[0080] The feature extraction layer is a standard convolutional neural network that uses the HR-NetV2p + FPN backbone network as the feature extractor for the abrasive chain image.

[0081] The region proposal layer is a lightweight convolutional neural network (RPN) used to find the abrasive feature regions.

[0082] The pooling layer adopts the strategy of augmented pooling to unify the size of the ROI regions.

[0083] The multi-task branch layer includes: the abrasive mask branch is a convolutional neural network, and both the abrasive classification branch and the abrasive bounding box branch are fully convolutional networks.

[0084] The abrasive chain images collected by online ferrography are successively subjected to downsampling and upsampling, and the upsampling results and downsampling results are fused to obtain the abrasive chain feature maps at the corresponding resolutions. The above steps are repeated to obtain four abrasive chain feature maps with different resolutions. The four abrasive chain feature maps are fused by adding them with weights of 4:3:2:1 from high to low resolution to obtain the fused abrasive chain feature map. The fused abrasive chain feature map is downsampled by 3×3 to construct a multi-level abrasive chain feature map. The RPN network is used to extract the ROI regions of the constructed multi-level abrasive chain feature map. The lengths and widths of all the extracted ROI regions are expanded to the smallest multiple of the pooling scale, and then the pooling operation is performed to obtain the ROI regions of a unified size. The ROI regions of the unified size are respectively input into the abrasive classification branch, the bounding box regression branch, and the mask branch to achieve the segmentation of the abrasive chain image.

[0085] Please refer to Figure 2 , in view of the characteristics of a large number of small-sized abrasives in the abrasive chain image, a feature extraction network with both high and low resolutions is constructed, and the feature extraction results are used as the input of the multi-task branch layer of the abrasive chain segmentation model;

[0086] S201. Adopt an abrasive feature extraction layer based on the high-resolution network HR-NetV2p, including a multi-resolution parallel branch, an abrasive chain image multi-resolution parallel branch fusion module, and an abrasive chain feature pyramid module;

[0087] S2011. Upsample the input abrasive chain image, and perform downsampling once every four upsamplings to gradually construct four parallel branches with resolutions from high to low, and extract the abrasive feature information at different resolutions;

[0088] Among them, the upsampling uses a 3×3 convolutional kernel with a stride of 1, and the downsampling uses a 3×3 convolutional kernel with a stride of 2.

[0089] S2012. The feature maps extracted from the four different-resolution branches from high to low are fused with weights of 4:3:2:1 and then downsampled by 3×3 to construct a multi-level abrasive grain feature map (feature pyramid).

[0090] S202. For the problem of different sizes of the candidate regions (ROIs) of the RPN, the strategy of augmented pooling is adopted to unify the ROI sizes;

[0091] S2021. To avoid pixel loss of the abrasive grain images during the pooling process, the lengths and widths of the bounding boxes of the candidate regions (ROIs) of different scales of the RPN are simultaneously expanded to the smallest multiple of the pooling scale, and the expanded scales are separately recorded in a matrix;

[0092] S2022. Normalize the expanded ROIs and use them as the input of the multi-task branch layer;

[0093] The multi-task branch layer includes an abrasive grain classification branch, an abrasive grain bounding box regression branch, and a mask branch.

[0094] S3. To improve the matching degree between the true position and the predicted position of the abrasive grains, define the loss functions of each task branch of the multi-task branch layer, and fuse them with weights as the loss function of the improved Mask-RCNN network, denoted as L loss ;

[0095] S301. Since the focus of the abrasive grain chain segmentation task is to ensure the full matching of the abrasive grain and mask position information, select the IOU loss function proposed based on confidence as the loss function of the ROI bounding box prediction branch, denoted as L box . The IOU loss function is proposed on the premise of considering the overlapping part between the predicted box and the true box. It calculates the error while retaining the correlation between the data, which not only meets the purpose of calculating the error of the loss function but also conforms to the actual situation;

[0096] The improved Mask R-CNN network selects the IOU loss function proposed based on confidence as the loss function L of the ROI bounding box prediction branch box as:

[0097] L box = 1 - IoU

[0098]

[0099] where A represents the predicted box, B represents the true box, and IoU reflects the overlapping part between the predicted box and the true box;

[0100] S302. Select the cross-entropy loss function to calculate the loss of the abrasive grain classification branch, denoted as L cls, and select the binary cross-entropy loss function to calculate the loss of the mask branch, denoted as \(L\). mask ;

[0101] S303. For the task of abrasive chain segmentation, there are only two categories, namely the background and the abrasive grains. The prediction accuracy of the bounding box and the mask will directly affect the accuracy of abrasive chain segmentation. Therefore, construct the following model loss function \(L\) that weights and fuses each task branch in parallel. loss ;

[0102] \(L\) loss = 0.2\(L\) cls + 0.4\(L\) box + 0.4\(L\) mask

[0103] where \(L\) cls is the loss function of the abrasive grain classification branch, \(L\) box is the loss function of the bounding box regression branch, and \(L\) mask is the loss function of the mask branch.

[0104] S4. Using the loss function \(L\) loss as the optimization objective, optimize the model using the abrasive chain segmentation dataset to achieve accurate segmentation of the abrasive chain image.

[0105] Using the loss function \(L\) loss as the optimization objective, input the abrasive chain segmentation dataset into the abrasive chain segmentation model based on the Mask-RCNN network architecture; evaluate the model performance every 2000 iterations and save the model parameters. After the training is completed, select the model parameters with the best performance; input the abrasive chain image to be segmented into the final model to obtain the abrasive grain mask image; locate the true position of the abrasive grains according to the abrasive grain mask information to achieve the task of segmenting individual abrasive grains from the abrasive chain image.

[0106] ADMD Iteration Algorithm Steps

[0107] t = t + 1

[0108]

[0109] m t = β1·m t-1 + (1 - β1)·g t

[0110]

[0111]

[0112]

[0113]

[0114] where t is the time step with an initial value of 0; g t - the gradient at time step t; θ is the parameter to be updated; f(θ) is the stochastic objective function of parameter θ; β1 is the exponential decay rate of the first moment; β2 is the exponential decay rate of the second moment; m t is the first moment estimate of the gradient; v t is the second moment estimate of the gradient; is the correction of the first moment estimate of the gradient; is the correction of the second moment estimate of the gradient; α is the learning rate; ε is a constant added to maintain numerical stability.

[0115] Set β1 = 0.9; β2 = 0.999; ε = 10 -8 ; α = 0.001.

[0116] In another embodiment of the present invention, a grinding particle chain segmentation system based on an improved Mask-RCNN network is provided. This system can be used to implement the above-mentioned grinding particle chain segmentation method based on an improved Mask-RCNN network. Specifically, the grinding particle chain segmentation system based on an improved Mask-RCNN network includes a data module, an improvement module, a loss module, and a segmentation module.

[0117] Among them, the data module extracts the overall contour of the grinding particle chain using the transmitted light image of the grinding particle chain, manually segments adjacent grinding particles according to the reflected light image of the grinding particle chain, superimposes them to obtain the single-grinding particle contour information in the grinding particle chain image, and constructs a grinding particle chain segmentation data set based on the single-grinding particle contour information;

[0118] The improvement module constructs a multi-task parallel deep learning model based on the Mask R-CNN framework;

[0119] The loss module defines the loss functions of each task branch in the multi-task branch layer, and weights and fuses the loss functions as the loss function L of the improved Mask-RCNN network loss ;

[0120] The segmentation module uses the loss function L obtained by the loss module loss as the optimization objective, optimizes the multi-task parallel deep learning model constructed by the improvement module using the grinding particle chain segmentation data set constructed by the data module to obtain the deep learning model with the optimal performance, and inputs the grinding particle chain image to be segmented into the deep learning model with the optimal performance to obtain the grinding particle mask image, realizing the precise segmentation of the grinding particle chain image.

[0121] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Generally, the components described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0122] Please refer to Figure 3 , using the HR-NetV2p network to extract the image feature information of the abrasive chain can ensure that the resolution of the abrasive chain image remains high during the operation process; using the RPN network to implement the extraction of the ROI region of the abrasive chain image and adopting the strategy of augmented pooling after the generation and screening of the ROI to provide guarantee for the segmentation accuracy of the abrasive chain; using three parallel branches to simultaneously complete the generation of the abrasive mask, abrasive classification, and abrasive bounding box regression tasks, which improves the model training speed and saves time.

[0123] Please refer to Figure 4 , the feature extraction module uses the HR-NetV2p network to fully retain the detailed information of the abrasive chain image during the convolution process; the ROI generation module extracts and screens the ROI regions belonging to the abrasive on the feature maps of different scales; the screened ROI regions are input into the multi-task branch to implement the prediction of the abrasive mask position, abrasive classification, and the prediction of the abrasive bounding box position.

[0124] Please refer to Figure 6 , the confidence IoU is the probability value that the Mask-RCNN network evaluates the feature region belonging to the target region. Y-pred represents the predicted box A, Y-truth represents the true box B, and IoU reflects the overlapping part between the predicted box and the true box. The definition formula is as follows:

[0125]

[0126] The confidence is obtained by jointly solving the center coordinates, length, and width of the predicted box and the true box, retaining the correlation between the values, conforming to the actual situation, and having scale invariance.

[0127] The IoU loss function proposed based on the confidence can ensure the relevance of the data while calculating the error between the predicted value and the true value. The definition formula is as follows:

[0128] IoU Loss =-ln(IoU)

[0129] Among them, IoU Loss is the IoU loss function value

[0130] Please refer to Figure 7 , the left side is the segmentation result of the abrasive chain, and the right side is the position of the abrasive mask. There are 20 abrasives in the figure. An abrasive chain segmentation method based on an improved Mask-RCNN network of the present invention successfully segments 19 abrasives, with an accuracy rate of 95%, and the position of the abrasive mask is fully matched with the actual position of the abrasive, providing a technical guarantee for the development of ferrography analysis technology.

[0131] In summary, an abrasive chain segmentation method and system based on an improved Mask-RCNN network of the present invention designs an abrasive chain segmentation method based on the HR-NetV2p network and Mask-RCNN network architecture, including establishing a loss function and proposing an augmented pooling strategy, which solves the problems of low intelligence and low segmentation accuracy of existing abrasive chain segmentation methods.

[0132] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0134] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1The functions specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in one Figure 1 one process or more processes and / or boxes Figure 1 step of the functions specified in one box or more boxes.

[0136] The above content is only to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention falls within the protection scope of the claims of the present invention.

Claims

1. An abrasive chain segmentation method based on an improved Mask-RCNN network, characterized in that It includes the following steps: S1. Extract the overall contour of the abrasive chain using the transmitted light image of the abrasive chain, manually segment adjacent abrasives based on the reflected light image of the abrasive chain, superimpose them to obtain the single abrasive contour information in the abrasive chain image, and construct an abrasive chain segmentation dataset according to the single abrasive contour information; S2. Construct a deep learning model with multi-task parallelism based on the Mask-RCNN framework. The deep learning model includes: A feature extraction layer, using the HR-NetV2p+FPN backbone network as the feature extractor for the abrasive chain image; A region proposal layer, which is a lightweight convolutional neural network for finding the abrasive feature regions; A pooling layer, adopting the strategy of augmented pooling to unify the size of the ROI regions; A multi-task branch layer, including an abrasive mask branch, an abrasive classification branch, and an abrasive bounding box branch. The abrasive mask branch is a convolutional neural network, and both the abrasive classification branch and the abrasive bounding box branch are fully convolutional networks; S3. Define the loss functions for each task branch of the multi-task branch layer, and weighted fuse the loss functions as the loss function for improving the Mask-RCNN network , specifically as follows: Among them, represents the loss function of the abrasive classification branch, is the loss function of the bounding box regression branch, represents the loss function of the mask branch; S4. Using the loss function obtained in step S3 as the optimization objective, training the multi-task parallel deep learning model constructed in step S2 with the abrasive chain segmentation dataset constructed in step S1, and then inputting the abrasive chain image to be segmented to obtain the abrasive mask image, so as to achieve the accurate segmentation of the abrasive chain image.

2. The abrasive chain segmentation method based on the improved Mask-RCNN network according to claim 1, characterized in that, Specifically, step S1 is as follows: Collect the reflected light image and transmitted light image of the abrasive chain, extract the contour of the abrasive chain in the transmitted light image using the Sobel operator, combine with manually marking the dividing line of adjacent abrasives based on the reflected light image to construct a complete abrasive chain segmentation label; repeat the above operations to establish an abrasive chain segmentation database, and divide the training set, validation set, and test set according to 8:1:1; use the training set as the input, verify the accuracy of the deep learning model through the validation set, and test the effect of the deep learning model through the test set.

3. The abrasive chain segmentation method based on the improved Mask-RCNN network according to claim 1, wherein In step S2, the upsampling of the feature extraction layer uses a 3×3 convolutional kernel with a stride of 1, and the downsampling of the feature extraction layer uses a 3×3 convolutional kernel with a stride of 2. ​ 4. The abrasive chain segmentation method based on the improved Mask-RCNN network according to claim 1, wherein In step S3, the loss function of the bounding box regression branch is as follows: Among them, A represents the predicted box, and B represents the ground truth box. It reflects the overlapping part between the predicted box and the ground truth box.

5. The abrasive chain segmentation method based on the improved Mask-RCNN network according to claim 1, characterized in that Specifically, step S4 is as follows: S401. Input the abrasive chain segmentation dataset into the deep learning model based on the Mask-RCNN network architecture; S402. Set the learning rate, the first-moment exponential decay rate, and the second-moment exponential decay rate of the ADAM optimizer; S403. Test the effect of abrasive chain segmentation once every n iterations and save the parameters of the deep learning model; S404. Screen out the model parameters with the optimal performance as the final result, input the abrasive chain image to be segmented into the trained deep learning model to obtain the abrasive mask image, and locate the true position of the abrasives according to the abrasive mask information to realize the segmentation of single abrasives from the abrasive chain image.

6. The abrasive chain segmentation method based on the improved Mask-RCNN network according to claim 5, characterized in that, In step S402, the learning rate is 0.001, the first-moment exponential decay rate is 0.9, and the second-moment exponential decay rate is 0.

999.

7. The abrasive chain segmentation method based on the improved Mask-RCNN network according to claim 5, characterized in that In step S403, n = 2000.

8. An abrasive chain segmentation system based on an improved Mask-RCNN network, characterized in that, It includes: A data module, which is used to extract the overall contour of the abrasive chain using the transmitted light image of the abrasive chain, manually segment adjacent abrasives based on the reflected light image of the abrasive chain, superimpose them to obtain the single abrasive contour information in the abrasive chain image, and construct an abrasive chain segmentation dataset according to the single abrasive contour information; An improvement module, which is used to construct a deep learning model with multi-task parallelism based on the Mask-RCNN framework. The deep learning model includes: A feature extraction layer, using the HR-NetV2p+FPN backbone network as the feature extractor for the abrasive chain image; A region proposal layer, which is a lightweight convolutional neural network for finding the abrasive feature regions; A pooling layer, adopting the strategy of augmented pooling to unify the size of the ROI regions; The multi-task branch layer includes an abrasive mask branch, an abrasive classification branch, and an abrasive bounding box branch. The abrasive mask branch is a convolutional neural network, and both the abrasive classification branch and the abrasive bounding box branch are fully convolutional networks; A loss module, which is used to define the loss functions of each task branch of the multi-task branch layer, and weighted fusion of the loss functions is used as the loss function for improving the Mask-RCNN network , specifically as follows: Among them, represents the loss function of the abrasive classification branch, is the loss function of the bounding box regression branch, represents the loss function of the mask branch; The segmentation module is used to take the loss function obtained by the loss module as the optimization objective, use the abrasive chain segmentation dataset constructed by the data module to optimize and improve the multi-task parallel deep learning model constructed by the module, then input the abrasive chain image to be segmented, and obtain the abrasive mask image to achieve accurate segmentation of the abrasive chain image.

Images