Agaricus bisporus phenotypic parameter digital characterization method based on instance segmentation model

Abstract

The invention discloses a digital characterization method for phenotypic parameters of agaricus bisporus based on an instance segmentation model. The method comprises the following steps: step 1, carrying out image acquisition, calibration and data set production on agaricus bisporus; step 2, constructing a YOLACT instance segmentation model, introducing a deformable convolution and a bidirectional feature pyramid into the YOLACT instance segmentation model for improvement so as to obtain an improved YOLACT instance segmentation model, and then training the improved YOLACT instance segmentation model by using a data set; step 3, utilizing the trained improved YOLACT instance segmentation model to segment mushrooms from the mushroom image and segment single mushroom caps from the adhered agaricus bisporus, and then designing a digital representation parameter calculation method to calculate and analyze phenotypic parameters of the mushroom caps; 4, after phenotype parameter data are obtained, the agaricus bisporus is graded according to the grading standard. The method can be applied to quality detection and quality grading in the pre-harvest stage of agaricus bisporus, and has the characteristics of rapidness, no damage and accuracy.

Description

Technical Field

[0001] The invention relates to the technical field of recognition of the external morphology of Agaricus bisporus, and in particular to a digital representation method of Agaricus bisporus phenotypic parameters based on an instance segmentation model. Background Art

[0002] The traditional method of cultivating Agaricus bisporus often uses a mushroom bed made of organic materials (agricultural waste such as straw, wood fiber and corn cobs), which is then inoculated with bacteria and placed in the mushroom bed for several weeks before the mushrooms begin to grow. However, this cultivation method has low production efficiency and is easily affected by weather changes, pollution and pests, so it is gradually being eliminated. With the development of technology, the cultivation of Agaricus bisporus is mostly carried out in a factory-based manner. By using sensors, computers, actuators and other equipment for precise ingredient mixing and three-stage tunnel fermentation, and precise monitoring and automatic control of mushroom production and collection, the quality and yield of Agaricus bisporus have been greatly improved. Although some of the work has been replaced by automated equipment, there are still several steps that require manual participation. At present, in the process of factory-based cultivation, the phenotypic parameters that affect the quality of Agaricus bisporus, such as the diameter size, roundness of the mushroom cap, whether there are spots on the surface of the mushroom cap and the color of the mushroom cap, are usually measured by manual inspection. The following defects exist in manual measurement or judgment based on experience:

[0003] (1) Due to the current labor shortage and the substantial increase in labor costs, manual inspection costs account for 15% to 30% of the total costs. Therefore, manual inspection methods consume a lot of manpower and financial resources.

[0004] (2) This method relies on human experience and knowledge. Subjective factors account for a large proportion of decision-making, resulting in large errors, which is not conducive to the refined control of the production process. In addition, it is impossible to accurately measure the quality of mushrooms and cannot meet the high standards of customers.

[0005] (3) Manual inspection is inefficient and cannot meet the needs of large-scale factory planting.

[0006] These defects have seriously restricted the development of the Agaricus bisporus industry. How to use automated detection technology to accurately identify Agaricus bisporus and extract its phenotypic parameters before mushroom harvesting is one of the keys to improving the efficiency of industrial Agaricus bisporus production and reducing production costs. Summary of the invention

[0007] The purpose of the present invention is to provide a method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model. The present invention can be applied to the quality detection and quality grading of Agaricus bisporus fruiting bodies before harvesting, and has the characteristics of being fast, non-destructive and accurate.

[0008] The technical solution of the present invention is a method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model, comprising the following steps:

[0009] Step 1: collect images, calibrate and prepare data sets for Agaricus bisporus;

[0010] Step 2: Based on the YOLACT instance segmentation model, deformable convolution and bidirectional feature pyramid are introduced into the YOLACT instance segmentation model to improve the model so as to improve the segmentation effect of the model, thereby obtaining an improved YOLACT instance segmentation model, and then the dataset is used to train the improved YOLACT instance segmentation model;

[0011] Step 3: Use the trained improved YOLACT instance segmentation model to segment mushrooms from mushroom images and separate individual mushroom caps from the adhered Agaricus bisporus, then design a digital representation parameter calculation method to calculate and analyze the phenotypic parameters of the mushroom caps;

[0012] Step 4: After obtaining the phenotypic parameter data, the Agaricus bisporus is graded according to the grading standard;

[0013] In step 2, the YOLACT instance segmentation model includes a backbone network, an FPN network, a prototype network, a prediction network, a non-maximum suppression processing module, a Crop processing module and a Threshold processing module; the process of introducing deformable convolution and bidirectional feature pyramid in the YOLACT instance segmentation model to improve the segmentation effect of the model is specifically: first, the unidirectional feature fusion pyramid of the FPN network is improved to a bidirectional feature pyramid. The bidirectional feature pyramid effectively fuses feature information at different levels by performing two bidirectional feature fusions, making the model more flexible and covering a larger area of ​​the learning target features, thereby improving the segmentation accuracy of the model; secondly, the deformable convolution is integrated into the backbone network. The deformable convolution captures more complex shapes and boundaries through the learned offset, so that the convolution kernel can more accurately extract the input image features, thereby improving the model's ability to recognize targets of different shapes.

[0014] In the above-mentioned method for digital characterization of Agaricus bisporus phenotypic parameters based on the instance segmentation model, in step 1, the image acquisition and data set preparation of Agaricus bisporus include building an Agaricus bisporus image acquisition platform, and calibrating and correcting the camera in the Agaricus bisporus image acquisition platform to ensure image quality; then a large number of Agaricus bisporus images of different specifications are collected to prepare a data set for improving the training of the YOLACT instance segmentation model.

[0015] In the aforementioned method for digitally representing Agaricus bisporus phenotypic parameters based on the instance segmentation model, the bidirectional feature pyramid performs feature fusion in a bottom-up and top-down manner:

[0016]

[0017] Where: Indicates bottom-up feature fusion; Convolution represents convolution operation; Upsample represents upsampling operation; P i Represents the i-th feature map in the bidirectional feature pyramid;

[0018]

[0019] Where: Indicates top-down feature fusion; Downsample indicates downsampling operation;

[0020] A weighting mechanism is applied to each feature map during fusion to adjust the influence of features at different scales. The weighted fusion process is expressed as follows:

[0021]

[0022] Where: α i and β i It is the learned weighting coefficient, which represents the weight of feature maps in different directions.

[0023] The aforementioned method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model, wherein the backbone network is a ResNet50 network, and the ResNet50 network is composed of five convolutional parts; the backbone network fused with the deformable convolution position is the last three parts of ResNet50.

[0024] In the aforementioned method for digitally characterizing Agaricus bisporus phenotypic parameters based on the instance segmentation model, the calculation process of the deformable convolution is as follows:

[0025]

[0026] Where: y(p0) represents the value of the output feature map at position p0; w(p n ) indicates p n The weight of the position; x(p0+p n +Δp n ) represents p0+p n +Δp n The pixel value at the position; Δp n Denotes the offset parameter; Δm k Indicates p n Position weight adjustment parameter, Δm k ∈[0, 1].

[0027] The aforementioned method for digitally representing Agaricus bisporus phenotypic parameters based on the instance segmentation model performs variable ROI pooling processing in the prediction network, and the variable ROI pooling processing is used to match the deformable convolution of the backbone network;

[0028] The sample output of the deformable ROI pooling for the kth pooled area is expressed as:

[0029]

[0030] Where: p kj represents the jth pixel position in the kth pooled area; Δp k represents the offset learned in the kth pooled region, n k represents the number of pixels in the kth pooled area; x(p kj +Δp k ) indicates p kj +Δp k Pixel value at position, Δm k Indicates p n Position weight adjustment parameter, Δm k ∈[0, 1].

[0031] The aforementioned method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model, wherein the phenotypic parameters of the mushroom cap include mushroom cap diameter, mushroom cap roundness, mushroom cap whiteness value, mushroom cap spot detection, and determining whether there are color spots and browning index on the mushroom cap surface;

[0032] The mushroom cap diameter is obtained by fitting the maximum circumscribed circle of the mushroom cap using the least square method, and then obtaining the diameter according to the maximum circumscribed circle;

[0033] The roundness of the mushroom cap is obtained by first obtaining the average distance from all contour points on the mushroom cap to the center of the maximum circumscribed circle and the deviation between the distance from the point on the mushroom cap contour to the center of the circle and the average distance, and then using the average distance and the deviation to express the roundness;

[0034] The mushroom cap whiteness value is obtained by converting a single Agaricus bisporus mushroom cap image after gamma correction from the RGB color space to the XYZ color space, and then using the CIE Ganz whiteness formula to accurately calculate the Agaricus bisporus whiteness value;

[0035] The spot detection of the mushroom cap is to first convert a single Agaricus bisporus mushroom cap image from the RGB color space into a grayscale image, then use a contour tracking algorithm to detect the edge contour of the object in each binary image to obtain spots in the grayscale image, then merge and remove duplicates of the spots, and finally extract the position and size of the spots according to the merged spots, and draw them in the original Agaricus bisporus mushroom cap image;

[0036] The method of judging whether there are spots on the mushroom cap surface is to calculate the deviation between the whiteness value of each pixel in the mushroom cap interest area and the average whiteness value, and reflect whether the color of each mushroom cap is uniform and whether there are spots according to the deviation.

[0037] In the aforementioned method for digitally representing Agaricus bisporus phenotypic parameters based on an instance segmentation model, in step 4, a BP neural network of a genetic algorithm is used to perform Agaricus bisporus grading.

[0038] The aforementioned method for digitally characterizing phenotypic parameters of Agaricus bisporus based on an instance segmentation model, the grading standard is a three-level standard, wherein the first-level standard is that the color of Agaricus bisporus is white and has no color spots, the cap shape is round or approximately round, and the cap diameter is greater than or equal to 5 cm; the second-level standard is that the color of Agaricus bisporus is white and has few color spots, the cap shape is round or approximately round, but there is slight deformity, and the cap diameter is 3.5-5 cm; the third-level standard is that the color of Agaricus bisporus is milky white and has color spots, the cap shape is round or approximately round, but there is deformity, and the cap diameter is less than 3.5 cm.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention has made in-depth improvements to the YOLACT instance segmentation algorithm, introducing deformable convolution (DCN) and bidirectional feature pyramid (BiFPN), which significantly improves the segmentation effect, especially in complex backgrounds covered with a large number of hyphae, enhancing the robustness of the model. DCN enables the convolution kernel to adaptively adjust its shape, more accurately capture the features of mushroom caps of different shapes, and adaptively respond to complex shape changes; BiFPN effectively integrates multi-scale information to enhance the segmentation accuracy of Agaricus bisporus of different sizes. The improved YOLACT model achieves accurate segmentation of adherent Agaricus bisporus, solves the problem of repeated segmentation of the same mushroom cap, and improves the average segmentation accuracy.

[0041] 2. After successfully segmenting the adhered Agaricus bisporus, the present invention further studies the accurate calculation of the diameter, roundness, browning index and whiteness of the Agaricus bisporus cap, and realizes the detection of color spots on the surface of the mushroom cap. When calculating the whiteness of the mushroom cap, spot detection is performed to remove the interference of soil on the surface of the mushroom cap, so that the result of the whiteness value is more accurate. Based on the whiteness value, the present invention also proposes an algorithm for judging whether there are color spots on the surface of the mushroom cap, and successfully realizes the detection of color spots of Agaricus bisporus. These achievements not only improve the accuracy of phenotypic parameter calculation, but also provide an effective method for the subsequent classification of phenotypic parameters of Agaricus bisporus.

[0042] 3. After comprehensively analyzing the phenotypic parameters of Agaricus bisporus, the present invention summarizes the phenotypic parameter-related data of each mushroom cap and implements classification according to national standards. The present invention optimizes the network weights based on the BP network and combines the genetic algorithm, significantly improving the overall performance of the improved BP network in the mushroom phenotypic parameter classification task, and conducts comparative analysis with the small sample network, aiming to find the best classification model in terms of accuracy and classification speed. Finally, the improved BP model was used to classify Agaricus bisporus, achieving a classification accuracy of more than 95%.

[0043] In summary, the present invention has successfully developed a set of digital measurement and grading methods for Agaricus bisporus phenotypic parameters, which provides a valuable reference for research in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the overall structure diagram of the improved YOLACT model of the present invention;

[0045] Figure 2 Schematic diagram of BiFPN network structure;

[0046] Figure 3 Schematic diagram of deformable convolution structure;

[0047] Figure 4 This is the structure diagram of the ResNet50 residual module after adding DCN;

[0048] Figure 5 It is a segmentation result diagram of the improved YOLACT instance segmentation model and Mask RCNN model of the present invention;

[0049] Figure 6 It is a segmentation result diagram of the improved YOLACT instance segmentation model of the present invention and the original model;

[0050] Figure 7 is a flow chart for digital characterization of Agaricus bisporus phenotypic parameters;

[0051] Figure 8 This is a schematic diagram of the maximum circumscribed circle of a single mushroom cap drawn in the original image;

[0052] Fig. 9 It is a schematic diagram of mushroom roundness calculation;

[0053] Fig.10 Schematic diagram of the method for calculating the roundness value of Agaricus bisporus mushroom cap according to the present invention;

[0054] Fig.11 The mushroom cap with soil attached and the detected and marked mushroom cap image;

[0055] Fig.12It is a schematic diagram of the threshold value for clustering color spots on the mushroom cap surface to distinguish between the two. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0057] Embodiment: A method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model comprises the following steps:

[0058] Step 1: collect images, calibrate and prepare data sets for Agaricus bisporus;

[0059] Step 2: Build a YOLACT instance segmentation model, introduce deformable convolution and bidirectional feature pyramid into the YOLACT instance segmentation model to improve the segmentation effect of the model, thereby obtaining an improved YOLACT instance segmentation model, and then use the dataset to train the improved YOLACT instance segmentation model;

[0060] Step 3: Use the trained improved YOLACT instance segmentation model to segment mushrooms from mushroom images and separate individual mushroom caps from the adhered Agaricus bisporus, then design a digital representation parameter calculation method to calculate and analyze the phenotypic parameters of the mushroom caps;

[0061] Step 4: After obtaining the phenotypic parameter data, the Agaricus bisporus is graded according to the grading standards.

[0062] In step 1 of this embodiment, the image acquisition and data set production of Agaricus bisporus includes building an Agaricus bisporus image acquisition platform, which mainly includes an industrial camera (Azure Kinect DK computer vision development kit released by Microsoft in 2020), a ring light source and a camera bracket, and the ring light source can be nested on the camera, and the two are installed on the bracket as a whole. Before starting the acquisition, the camera is calibrated and corrected to restore the accurate relationship between the 3D coordinates in the real world and the 2D projection in the camera-captured image to ensure image quality. Camera calibration is completed by using a calibration plate with a specific pattern and taking pictures of it at multiple different positions and angles, and finally converting the world coordinate system into an image coordinate system. In this embodiment, Zhang's calibration method is used to obtain camera parameters.

[0063] The images of Agaricus bisporus in this embodiment are all collected in the modern Agaricus bisporus mushroom house of Longchen Modern Agricultural Technology Co., Ltd. in Tongxiang City, Jiaxing City, Zhejiang Province. The collected Agaricus bisporus images are numbered and marked and displayed in different colors using ISAT annotation software. Secondly, selective annotation is performed during annotation. The annotation principle is to only mark the complete mushroom caps, and not mark the incomplete mushroom caps and the mushroom caps that are too small. After each picture is annotated, a corresponding json file will be generated, which contains the picture name, the number of each mushroom cap, and the segmentation area information of each mushroom cap. Finally, the prepared data set is divided. There are a total of 1,400 Agaricus bisporus data sets in this study, and the data set is divided into training set, verification set and test set according to 8:1:1. The total number of single mushroom samples in the training set exceeds 10,000, and the data set is rich in variety, meeting the needs of deep learning.

[0064] In step 2 of this embodiment, the YOLACT instance segmentation model includes a backbone network, an FPN network, a prototype network, a prediction network, a non-maximum suppression processing module, a Crop processing module and a Threshold processing module.

[0065] The backbone network usually refers to the main part of the entire deep neural network, which is responsible for extracting the features of the input data. The backbone network is usually composed of multiple convolutional layers or other feature extraction layers, which are used to gradually extract the features of the input data. In image processing tasks, the backbone network is usually used to extract global and local features of the image, such as edges, textures, and shapes. The image size of the YOLACT model input is 550×550, and the backbone network used is ResNet50, which consists of 5 convolution parts, from Conv1 to Conv5 corresponding to C1 to C5 in the YOLACT model. The reason for using ResNet as the backbone network is that it has a residual module, which can retain more image features and effectively solve problems such as gradient explosion during model training. ResNet adds cross-layer connections, which can directly pass input data across layers for equal mapping, and then add it to the result of the convolution operation. When the number of input and output channels is the same, the input image is x, the output is H(x), and the output after convolution is a nonlinear function of F(x). The final output is H(x)=F(x)+x. The residual refers to the "difference", that is, F(x), and the network is transformed into a residual function F(x)=H(x)-x. In this way, the residual function is easier to optimize than F(x)=H(x). When the number of input and output channels is different, a 1×1 convolution layer is added to the cross-layer connection to adjust the input dimension so that the input and output dimensions are the same. The entire backbone network transmits the Agaricus bisporus image as input data layer by layer. First, the Agaricus bisporus image is preprocessed and the size is adjusted from 550×550×3 to 224×224×3 to reduce the computational complexity of the network. After preprocessing, the input Agaricus bisporus image is downsampled by 7×7 convolution, while retaining the original image information as much as possible. After downsampling, BatchNorm is used for normalization to reduce the amount of calculation and increase the convergence speed of the network, thereby improving the performance of the model structure. After normalization, the activation function (ReLU) is used to increase the nonlinear ability of the model, while reducing the model parameters to speed up the model training. The last step is maximum pooling. The main purpose of maximum pooling is to reduce the size and number of parameters of the feature map and simplify the network complexity. After maximum pooling, the contour features of the Agaricus bisporus image are extracted in sequence through the C2-C5 convolution blocks. The feature extraction part of the ResNet50 model is composed of 16 residual modules. Compared with other models, its network structure is deeper, so it can extract more Agaricus bisporus image features.

[0066] FPN network (Feature Pyramid Networks) is a deep learning model used in target detection, semantic segmentation and other fields. The Feature Pyramid Network uses a top-down and lateral connection method to fuse feature maps at different scales. The top-down method is similar to the traditional convolutional neural network, which gradually reduces the feature map from the high level to the low level; the lateral connection is to fuse the feature maps of different scales at the same level, so that the feature information at different scales can be shared. The FPN network obtains P5 from C5 through one convolution layer, and then uses bilinear interpolation to double the feature map of P5, and adds it to the convolution C4 to obtain P4, and then uses the same method to obtain P3. Then, P5 is convolved and downsampled to obtain P6, and P6 is convolved and downsampled to obtain P7, thereby establishing the FPN network. Finally, P3 is sent to the prototype network (Protonet), and P3-P7 are also sent to the prediction network (Prediction Head) at the same time. The advantage of using FPN is that the model learns richer features and is more conducive to segmenting target objects of different sizes.

[0067] The prototype network is the network branch of the YOLACT instance segmentation model that generates the prototype mask. It is used to process the input feature map to generate the prototype mask. With the P3 feature map as input, Protonet generates k 138×138 prototype masks. There are two categories for Agaricus bisporus, one is the mushroom to be segmented, and the other is its background. Therefore, it is a binary classification task (k=2 at this time), so Protonet finally generates 2 138×138 prototype masks.

[0068] The prediction network (Prediction Head) is used to generate three main outputs: detection box location coordinates (BoxLocation), mask coefficients (Mask Coefficients) and category confidence (Class Confidence), where the location coordinate output dimension is: [None, W*H*a, 4], None represents the batch size, W and H are the width and height of the feature map, a is the number of anchor boxes at each position, usually 3 (corresponding to different aspect ratios 1:1, 1:2 and 2:1); mask coefficient output dimension: [None, W*H*a, k], k is the number of categories; confidence output dimension: [None, W*H*a, k]. Taking P3 as an example, assuming that the dimension of P3 is W*H*256, then its number of anchors is a3=W*H*3. Next, Prediction Head generates three types of outputs: Since there are 2 categories in the Agaricus bisporus dataset (including background), the category confidence dimension is a3*2; the position offset dimension is a3*4; and the mask coefficient dimension is a3*2. The operations on P4-P7 are the same. Finally, these results are concatenated, marked a=a3+a4+a5+a6+a7, to obtain the total category confidence. Since there are 2 categories in the Agaricus bisporus dataset (including background), its dimension is a*2; all position offsets, the dimension is a*4; all mask confidences, the dimension is a*2.

[0069] Non-maximum suppression processing module (NMS), in the YOLACT instance segmentation model, the output of the Prediction Head network will be processed by non-maximum suppression (NMS). Prediction Head generates a large number of anchor boxes, where the position information of each anchor box is adjusted by the position offset to determine the final ROI (Region of Interest) position. Since these ROIs may overlap, the non-maximum suppression processing module (NMS) is used to filter out the optimal anchor box and remove redundant overlapping boxes. The processing results of NMS are combined with the k 138*138 Proto prototypes output by Protonet for operation (superposition, cropping, threshold segmentation) to obtain the final detection result.

[0070] The Crop processing module and the Threshold processing module are the key steps to combine the anchor box processed by NMS with the prototype mask output by Protonet into the final detection result. The function of the Crop processing module is to crop the weighted mask to the area of ​​the detection box (anchor box) so that it can be accurately aligned with the target. After NMS processing, a set of optimal anchor boxes are obtained, and each anchor box represents the final detection box after position regression adjustment. The format of these boxes is usually (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner and (x2, y2) is the coordinate of the lower right corner, and accurate cropping is performed through the coordinates.

[0071] The function of the Threshold processing module is to convert the cropped mask into a binary mask to clarify the boundary of the target area. Apply the threshold operation to the cropped mask image to convert it into a binary image. The basic idea of ​​the threshold operation is to select a suitable threshold to divide the pixel values ​​in the mask image into foreground (target area) and background.

[0072] Based on the above detailed introduction to the YOLACT instance segmentation model, the present invention improves the YOLACT instance segmentation model from the FPN network structure and the backbone network in order to obtain an improved YOLACT instance segmentation model.

[0073] First, by modifying the FPN network of the YOLACT model, the unidirectional feature fusion pyramid (FPN) is improved to a bidirectional feature pyramid (BiFPN), making the model receptive field more flexible, the coverage area of ​​the learning target features larger, and more suitable for the segmentation of Agaricus bisporus instances, thereby improving the segmentation accuracy of the model. By performing two bidirectional feature fusions, feature information at different levels can be effectively fused, thereby improving the detection performance of the algorithm. Secondly, a deformable convolution (DCN) is added to the backbone network so that the convolution kernel can more accurately extract the input image features, thereby improving the network's ability to recognize targets of different shapes. Since traditional convolution operations are usually fixed on a regular grid, deformable convolution can capture more complex shapes and boundaries through learned offsets. This enables the network to perform better when dealing with irregular object boundaries or curved edges. The overall structure of the improved YOLACT model is shown in the figure below. Figure 1 shown.

[0074] Specifically, the network structure of BiFPN consists of multiple BiFPN blocks and multiple convolutional layers, where the BiFPN block includes two feature fusion modules with attention mechanisms, which can effectively fuse the feature information of multiple layers. During the training process, BiFPN uses the Focal Loss loss function. By introducing an adjustment factor, the model pays more attention to difficult samples (samples that are difficult to classify) during training, which can effectively solve the problem of imbalance between positive and negative samples. At the same time, BiFPN can also be used multiple times. The BiFPN network used in this embodiment is as follows: Figure 2 As shown, the BiFPN performs feature fusion in a bottom-up and top-down manner:

[0075]

[0076] Where: Indicates bottom-up feature fusion; Convolution represents convolution operation; Upsample represents upsampling operation; P i Represents the i-th feature map in the bidirectional feature pyramid;

[0077]

[0078] Where: Indicates top-down feature fusion; Downsample indicates downsampling operation;

[0079] A weighting mechanism is applied to each feature map during fusion to adjust the influence of features at different scales. The weighted fusion process is expressed as follows:

[0080]

[0081] Where: α i and β i It is the learned weighting coefficient, which represents the weight of feature maps in different directions.

[0082] Deformable Convolution (DCN) is a new type of convolutional neural network module that allows the convolution kernel to deform during the convolution process to better adapt to local changes in the input data. Figure 3 Schematic diagram of the deformable convolution structure in this embodiment. In this embodiment, the original 3×3 convolution in the residual module of the backbone network ResNet50 is replaced with a DCN module, and the spatial sampling position of the additional offset is added to the original convolution, so that the convolution kernel can adapt to the different deformations of the mushroom cap according to the actual situation, and more accurately extract the input features, thereby improving the network's ability to identify deformed targets. The structure diagram of the ResNet50 residual module after adding DCN is shown in the figure. Figure 4As shown. After adding the DCN module, the need to learn additional offsets will increase the number of model parameters, and not all convolutions will be replaced by deformable convolutions to improve the segmentation accuracy of the model. In order to balance the accuracy and model parameters, the present invention adjusts and analyzes the position of ResNet50 fusion DCN, and finally determines to add the DCN module to the last three layers of ResNet50. Because the first two layers of the model extract a lot of feature information, adding the DCN module may not be ideal and may increase the complexity of the model. After the model extracts some semantic features and then adds deformable convolution, the learned features will become better, and the increase in model parameters is avoided. In this embodiment, for deformable convolution, it does not directly change the shape of the convolution kernel, but introduces an offset to modify the sampling result on the basis of ordinary convolution, thereby indirectly achieving the effect of changing the shape of the convolution kernel. First, define the region R as a regular rectangular area in the input feature map x. If each element at each position in R has an offset (Δp learned by the neural network n |n=1,...,N), where N is the number of elements in the convolution kernel, then the deformable convolution is calculated as:

[0083]

[0084] Where: y(p0) represents the value of the output feature map at position p0; w(p n ) indicates p n The weight of the position; x(p0+p n +Δp n ) represents p0+p n +Δp n The pixel value of the position;

[0085] Although, the offset parameter Δp n (x direction and y direction respectively) can be learned by adding additional branches to the input feature map, and the geometric transformation ability of the model is enhanced, but there are still some problems. For example, when sampling, if the background area is sampled, it will affect the target classification kernel detection, so the contribution of the background area should be suppressed. Therefore, the present invention changes the number of output channels from 2N to 2N+1, and activates the last dimension to (0,1) with Sigmoid, and uses Δm k It means the contribution of the current pixel. If it is background noise, then Δm k tends to 0, so the deformable convolution in the above formula becomes:

[0086]

[0087] To match the deformable convolution, the size and position of the window should be dynamically adjusted during the deformable ROI pooling process to better capture the spatial changes and details of the target.

[0088] The sample output of the deformable ROI pooling for the kth pooled area is expressed as:

[0089]

[0090] Where: p kj represents the jth pixel position in the kth pooled area; Δp k represents the offset learned in the kth pooled region, n k represents the number of pixels in the kth pooled area; x(p kj +Δp k ) indicates p kj +Δp k The pixel value at the location.

[0091] The improved YOLACT instance segmentation model designed by the present invention combining BiFPN and DCN modules is evaluated, and the performance of the proposed Agaricus bisporus instance segmentation model is verified using a special Agaricus bisporus dataset. At the same time, in order to verify whether the model improvement strategy is effective, an ablation experiment is set to verify whether its effect is excellent. Finally, a comparative experiment verifies the effectiveness of the improved YOLACT instance segmentation model.

[0092] The improved YOLACT instance segmentation model is compared with the original model, using the average precision (AP), the average precision value of different IOUs (mAP) and the backbone network parameters (Parameters, Params) as evaluation indicators. By controlling the variable method, the segmentation comparison experiment is carried out on the same network environment and data set. The results are shown in Table 1:

[0093]

[0094]

[0095] Table 1

[0096] It can be concluded from Table 1 that the evaluation indicators of the improved YOLACT instance segmentation model have been significantly improved, and the average segmentation accuracy has increased by 4.04%. The model with the addition of BiFPN and DCN modules has significantly improved the segmentation accuracy of the adhered mushroom caps, proving that the improved model has a good segmentation effect on adhered Agaricus bisporus. Although the number of parameters and the amount of calculation of the improved model have increased slightly, the cost is not large relative to the improved segmentation accuracy.

[0097] In order to verify the segmentation performance of the YOLACT model after fusing the BiFPN and DCN modules, the segmentation results of the improved YOLACT instance segmentation model on the adhered Agaricus bisporus were compared with the segmentation results of the original model and the Mask RCNN model. The experimental segmentation results are shown in the figure below. Figure 5 shown. Figure 5 (a) in the figure is the segmentation result of the Mask RCNN model. Figure 5 (b) is the segmentation result diagram of the improved YOLACT instance segmentation model of the present invention. From the comparison of the circled area, it can be seen that the Mask RCNN model is not accurate in segmenting the edge of the mushroom cap, while the improved YOLACT instance segmentation model after integrating BiFPN and DCN accurately segments the edge of the mushroom cap. The improved segmentation accuracy is significantly improved, and the interference with small target mushroom bags is also significantly improved. There are many small information of target features in the original backbone network that are very easy to be segmented. Adding DCN can increase the receptive field of the model, increase the model's ability to perceive details, and improve the accuracy of the model's target segmentation. Figure 6 (a) is the result of the original YOLACT model segmentation; Figure 6 (b) is the result of the improved YOLACT instance segmentation model segmentation. It can be seen that the improved YOLACT model better identifies each Agaricus bisporus individual in the image and accurately marks the segmentation accuracy of each mushroom cap. By integrating the BiFPN and DCN modules, the model feature extraction capability is enhanced, solving the problem of repeated segmentation of the same mushroom cap.

[0098] Furthermore, an ablation test was conducted on the homemade Agaricus bisporus dataset to determine the effectiveness of each improvement strategy. BiFPN and DCN were added to the original model in turn. “√” indicates improvement and “×” indicates no improvement. The results are shown in Table 2:

[0099]

[0100] Table 2

[0101] After the introduction of BiFPN, the network learns more feature information and more accurately extracts the complete features of the mushroom cap. The average accuracy of the model is improved by 2.54%. When the threshold is 0.5, the average precision mean mAP is 95.83%, and when the threshold is 0.75, the average precision mean mAP is 95.83%, which is significantly improved compared with the original YOLACT model. After the introduction of the DCN module, many semantic information that is easily segmented can be paid attention to, and the feature information of the most important part of the perception data can be perceived. After the introduction of the DCN module, the average segmentation accuracy is 89.30%, which is 1.45% higher than the original YOLACT model, and the segmentation effect of the edge contour of the mushroom cap is also greatly improved. Therefore, the above results verify that both improved methods of the present invention can effectively improve the segmentation accuracy of the model, and improve the problems of low segmentation accuracy, target misdetection, and missed detection of the adhesion Agaricus bisporus caused by the planting environment. The average segmentation accuracy of the improved YOLACT instance segmentation model with BiFPN and DCN added at the same time is increased to 91.89%, which is significantly improved by 4.04% compared with the average accuracy of the original YOLACT model. Experimental results show that both of our improved schemes can effectively improve the segmentation accuracy of the model, verifying the effectiveness of the method proposed in this paper.

[0102] In step 3 of this embodiment, the phenotypic parameters of Agaricus bisporus are calculated based on machine vision and image processing technology, and the phenotypic parameters are digitally represented. The process is as follows: Figure 7 The phenotypic parameters of the mushroom cap include the mushroom cap diameter, the mushroom cap roundness, the mushroom cap whiteness value, the mushroom cap spot detection, and the judgment of whether there are color spots and browning index on the mushroom cap surface.

[0103] The mushroom cap diameter is obtained by fitting the maximum circumscribed circle of the mushroom cap using the least square method, and then obtaining the diameter according to the maximum circumscribed circle. The maximum circumscribed circle result of the fitted single mushroom cap is drawn in the original image, such as Figure 8 The image shown is after fitting and drawing the results. The circle in the figure is the circle fitted by this algorithm according to the mushroom cap contour, and then the required diameter is converted into a real value by spatial resolution.

[0104] The roundness of the mushroom cap is obtained by first obtaining the average distance from all contour points on the mushroom cap to the center of the maximum circumscribed circle and the deviation between the distance from the point on the mushroom cap contour to the center of the circle and the average distance, and then using the average distance and the deviation to express the roundness; Fig. 9The diagram of mushroom roundness calculation is shown in the figure. The circle drawn by the solid line is the maximum circumscribed circle fitted by the least square method, the circle drawn by the dotted line is the circle with the average distance D from all points on the mushroom cap contour to the center as the radius, the area drawn by the curve is the contour of the mushroom cap, and Error is the deviation between the actual distance from the point on the mushroom cap contour to the center and the average distance D. According to the national standard (GB / T 23190-2008), the closer the shape of Agaricus bisporus is to a perfect circle, the higher its quality, and vice versa. Therefore, it is of great significance to quickly and accurately judge the appearance of each mushroom cap. The present invention calculates the roundness of 766 individual mushroom caps according to the proposed roundness algorithm. In order to more intuitively demonstrate the reliability of the algorithm, as shown in the figure. Fig.10 The figure shows the roundness value of the mushroom cap of Agaricus bisporus calculated according to the method of the present invention. The black part in the figure is the mask of a single mushroom cap obtained by segmentation. From the results in the figure, it can be seen that the closer the shape of the mushroom cap is to a circle, the larger the roundness value is, and the more serious the shape distortion is, the smaller the roundness value is.

[0105] The mushroom cap whiteness value is obtained by converting a gamma-corrected single Agaricus bisporus mushroom cap image from an RGB color space to an XYZ color space, and then using the CIE Ganz whiteness formula to accurately calculate the Agaricus bisporus whiteness value; specifically, first, using a D65 light source as the light source of an image acquisition system to obtain an RGB image, normalizing the three RGB color channels, scaling them to 0 to 1, and after scaling, using gamma to correct the normalized RGB to obtain the corrected sRGB, converting a gamma-corrected single Agaricus bisporus mushroom cap image from an RGB color space to an XYZ color space, and then using the CIE Ganz whiteness formula to accurately calculate the Agaricus bisporus whiteness:

[0106] First, calculate the whiteness value of each pixel on Agaricus bisporus:

[0107] W i =Y+800×(mx i )+1700×(ny i );

[0108] Where: Y is the brightness factor of the ith pixel on Agaricus bisporus, x i and i are the brightness factor and chromaticity coordinates of the ith pixel on Agaricus bisporus, and m and n are the chromaticity coordinates of a standard white plate under the CIE standard D65 illuminant viewing angle, which are 0.3138 and 0.3309, respectively.

[0109] After the whiteness values ​​of all pixels on the mushroom cap are calculated, the whiteness value of a single mushroom cap is:

[0110]

[0111] Wherein, N is the total number of mushroom cap pixels, Wi (i=1, 2, 3, ..., N) is the whiteness value of the i-th pixel, and W is the average whiteness value of the mushroom cap.

[0112] There are interference factors in the actual calculation. After the mushrooms grow, the mushroom caps may be attached with dirt on their surface. Therefore, directly calculating the whiteness value of the mushroom caps may produce deviations. Therefore, it is necessary to detect the spots first. The steps for spot detection of the mushroom caps are as follows:

[0113] (1) First, the input single image is converted from the RGB color space to a grayscale image. Then, the threshold range T1, T2 and the step size S are set to convert the grayscale image into multiple threshold binary images. Specifically, the first threshold is T1, the second threshold is T1+S, and so on, until T2 is reached.

[0114] (2) Contour detection and connected region extraction: Use the contour tracking algorithm proposed by Suzuki to detect the edge contours of objects in each binary image. Then, through contour tracking, identify the connected pixel sets in each binary image. These areas are called connected regions, which usually represent spots or targets in grayscale images. The connected regions detected in all binary images are stored in a Blobs list.

[0115] (3) Spot merging and deduplication: Since all the spot connected regions obtained in step (2) are stored in a list, there may be a large number of overlapping and adjacent spot connected regions between spots. Therefore, it is necessary to merge the adjacent and overlapping spot connected regions. Calculate the geometric center and the maximum circumscribed circle radius of each region, set the distance threshold Tb (the optimal value is obtained through experiments later), and then there may be the following four relationships between two spots: ① If the distance is less than Tb and the sum of the radii of Kp1 and Kp2 is less than or equal to the Euclidean distance between the center coordinates of the two spots

[56] , then the two spot connected regions are only adjacent but not overlapping; ② If the distance is less than Tb and the sum of the radii of Kp1 and Kp2 is greater than the distance between the center coordinates of the two spots, then the two spot connected regions are adjacent and overlapping; ③ If the distance is greater than Tb and the sum of the radii of Kp1 and Kp2 is greater than or equal to the distance between the center coordinates of the two spots, then the two spot connected regions overlap but are not adjacent; ④ If the distance is greater than Tb and the sum of the radii of Kp1 and Kp2 is less than or equal to the distance between the center coordinates of the two spots, then the two spot connected regions are neither adjacent nor overlapping. All the spot regions in the list are traversed in pairs. If the two spot regions meet the conditions ①②③, the regions are merged. If they meet the condition ④, the spot regions are retained. The merging process is to mark the two spot regions as the same region, calculate the average of the center coordinates of the two spot regions as the center coordinates of the merged new region, and the sum of the radius of the two spot regions plus half of the sum of the distance between the center coordinates of the two is the radius of the new region. After the new region is formed, the original two regions in the list are replaced by the new region, and the traversal and update of the merged regions are continued in pairs until all the spot regions in the list meet only the condition ④, then the merging is stopped.

[0116] (4) Spot drawing: Finally, the position and size of the spots are extracted based on the connected domain of the merged spots and drawn in the original image. This information will be used for subsequent quality analysis and whiteness value calculation to ensure the accuracy of the calculation results. According to the above steps, the information of the grayscale image spots is obtained, and the detected spot area is not included in the calculation of the mushroom cap whiteness, which can greatly reduce the error of the whiteness value calculation, such as Fig.11 Shown are mushroom caps with mud attached and detected and marked mushroom cap images, where (a), (b), (c), (d) are mushroom caps with mud attached to the surface; (e), (f), (g), (h), (i) are the results detected and marked by our algorithm.

[0117] The method of judging whether there are color spots on the mushroom cap surface is to calculate the deviation between the whiteness value of each pixel in the mushroom cap interest area and the average whiteness value, and reflect whether the color of each mushroom cap is uniform and whether there are color spots according to the deviation. The calculation formula for whether there are color spots on the mushroom cap surface is as follows:

[0118]

[0119] In order to more accurately determine whether there are spots on the surface of the mushroom cap, it is necessary to determine the threshold for distinguishing the two. 70 mushroom caps with spots and 70 mushroom caps without spots are selected respectively, and then the clustering idea is used to find the threshold for distinguishing the two. Fig.12 As shown in the figure, when the Error value is greater than 0.21, it is judged that there are color spots on the surface of the mushroom cap, and the larger the value, the larger the area of ​​the surface color spots; when the Error value is less than 0.21, there are no color spots on the surface of the mushroom cap, and the smaller the value, the smoother the surface and the more uniform the color.

[0120] The above phenotypic parameters are calculated based on the systematic and comprehensive nature of the national standard (GB / T 23190-2008), which can automatically characterize the phenotypic parameters of Agaricus bisporus and provide a more accurate and efficient tool for Agaricus bisporus production and quality assessment.

[0121] In step 4 of this embodiment, the phenotypic parameters extracted from the Agaricus bisporus image are classified, as shown in Table 3:

[0122]

[0123] Table 3

[0124] Establishing a grading model for Agaricus bisporus can effectively reduce manual judgment errors, improve evaluation speed, and realize accurate evaluation of mushroom specifications and grades. The model is constructed based on the phenotypic parameters (roundness, diameter, whiteness of mushroom cap and whether there are spots) extracted from the Agaricus bisporus image. This embodiment uses a BP neural network of a genetic algorithm to perform the grading of Agaricus bisporus. The BP neural network is a multi-layer feedforward network trained by error back propagation (abbreviated as error back propagation). Its basic idea is the gradient descent method, which uses the gradient search technology to minimize the error mean square error between the actual output value and the expected output value of the network. In view of the many problems of the BP network, in order to improve the accuracy of the grading of Agaricus bisporus, the genetic algorithm is combined with the BP network to improve and optimize the BP network. The genetic algorithm screens individuals (a network weight and bias) according to the fitness function through selection, crossover and mutation in inheritance. Individuals with good fitness values ​​are retained, individuals with poor fitness values ​​are eliminated, and new individuals inherit the information of the previous generation and are better than the previous generation. This cycle is repeated until the conditions are met (such as the fitness reaches the target value and the maximum number of generations is reached). The BP neural network based on genetic algorithm can be used to grade Agaricus bisporus according to the grading standards, which reduces labor costs and greatly improves production efficiency.

[0125] In summary, the present invention will segment Agaricus bisporus based on the YOLACT instance segmentation model, calculate the diameter, roundness and whiteness of the mushroom cap, and introduce spot detection technology to improve the accuracy of whiteness calculation. The present invention realizes the rapid and efficient extraction of Agaricus bisporus phenotypic parameters, fills the gap of detailed calculation and digital evaluation of Agaricus bisporus phenotypic parameters, provides a scientific basis for the production and quality control of modern factory-based Agaricus bisporus cultivation, and provides a reference for the application of related automated detection technologies.

Claims

1. A digital characterization method for Agaricus bisporus phenotypic parameters based on an instance segmentation model, characterized in that: The steps include: Step 1: collect images, calibrate and prepare data sets for Agaricus bisporus; Step 2: Based on the YOLACT instance segmentation model, deformable convolution and bidirectional feature pyramid are introduced into the YOLACT instance segmentation model to improve the model so as to improve the segmentation effect of the model, thereby obtaining an improved YOLACT instance segmentation model, and then the dataset is used to train the improved YOLACT instance segmentation model; Step 3: Use the trained improved YOLACT instance segmentation model to segment mushrooms from mushroom images and separate individual mushroom caps from the adhered Agaricus bisporus, then design a digital representation parameter calculation method to calculate and analyze the phenotypic parameters of the mushroom caps; Step 4: After obtaining the phenotypic parameter data, the Agaricus bisporus is graded according to the grading standard; In step 2, the YOLACT instance segmentation model includes a backbone network, an FPN network, a prototype network, a prediction network, a non-maximum suppression processing module, a Crop processing module and a Threshold processing module; the process of introducing deformable convolution and bidirectional feature pyramid in the YOLACT instance segmentation model to improve the segmentation effect of the model is specifically: first, the unidirectional feature fusion pyramid of the FPN network is improved to a bidirectional feature pyramid. The bidirectional feature pyramid effectively fuses feature information at different levels by performing two bidirectional feature fusions, making the model more flexible and covering a larger area of ​​the learning target features, thereby improving the segmentation accuracy of the model; secondly, the deformable convolution is integrated into the backbone network. The deformable convolution captures more complex shapes and boundaries through the learned offset, so that the convolution kernel can more accurately extract the input image features, thereby improving the model's ability to recognize targets of different shapes.

2. The method for digital characterization of Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 1, characterized in that: In step 1, the image acquisition and data set preparation of Agaricus bisporus include building an Agaricus bisporus image acquisition platform, and calibrating and correcting the camera in the Agaricus bisporus image acquisition platform to ensure image quality; then collecting a large number of Agaricus bisporus images of different specifications to prepare a data set for improving the training of the YOLACT instance segmentation model.

3. The method for digital characterization of Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 1, characterized in that: The bidirectional feature pyramid performs feature fusion in a bottom-up and top-down manner: Where: Indicates bottom-up feature fusion; Convolution represents convolution operation; Upsample represents upsampling operation; P i Represents the i-th feature map in the bidirectional feature pyramid; Where: Indicates top-down feature fusion; Downsample indicates downsampling operation; A weighting mechanism is applied to each feature map during fusion to adjust the influence of features at different scales. The weighted fusion process is expressed as follows: Where: α i and β i It is the learned weighting coefficient, which represents the weight of feature maps in different directions.

4. The method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 3, characterized in that: The backbone network is a ResNet50 network, which consists of five convolutional parts; the backbone network fused with deformable convolution positions is the last three parts of ResNet50.

5. The method for digital characterization of Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 4, characterized in that: The calculation process of the deformable convolution is as follows: Where: y(p0) represents the value of the output feature map at position p0; w(p n ) indicates p n The weight of the position; x(p0+p n +Δp n ) represents p0+p n +Δp n The pixel value at the position; Δp n Denotes the offset parameter; Δm k Indicates p n Position weight adjustment parameter, Δm k ∈[0, 1].

6. The method for digitally characterizing Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 5, characterized in that: Variable ROI pooling is performed in the prediction network to match the deformable convolution of the backbone network; The sample output of the deformable ROI pooling for the kth pooled area is expressed as: Where: p kj represents the jth pixel position in the kth pooled area; Δp k represents the offset learned in the kth pooled region, n k represents the number of pixels in the kth pooled area; x(p kj +Δp k ) indicates p kj +Δp k Pixel value at position, Δm k Indicates p n Position weight adjustment parameter, Δm k ∈[0, 1].

7. The method for digital characterization of Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 1, characterized in that: The phenotypic parameters of the mushroom cap include mushroom cap diameter, mushroom cap roundness, mushroom cap whiteness value, mushroom cap spot detection, and judging whether there are color spots and browning index on the surface of the mushroom cap; The mushroom cap diameter is obtained by fitting the maximum circumscribed circle of the mushroom cap using the least square method, and then obtaining the diameter according to the maximum circumscribed circle; The roundness of the mushroom cap is obtained by first obtaining the average distance from all contour points on the mushroom cap to the center of the maximum circumscribed circle and the deviation between the distance from the point on the mushroom cap contour to the center of the circle and the average distance, and then using the average distance and the deviation to express the roundness; The mushroom cap whiteness value is obtained by converting a single Agaricus bisporus mushroom cap image after gamma correction from the RGB color space to the XYZ color space, and then using the CIE Ganz whiteness formula to accurately calculate the Agaricus bisporus whiteness value; The spot detection of the mushroom cap is to first convert a single Agaricus bisporus mushroom cap image from the RGB color space into a grayscale image, then use a contour tracking algorithm to detect the edge contour of the object in each binary image to obtain spots in the grayscale image, then merge and remove duplicates of the spots, and finally extract the position and size of the spots according to the merged spots, and draw them in the original Agaricus bisporus mushroom cap image; The method of judging whether there are spots on the mushroom cap surface is to calculate the deviation between the whiteness value of each pixel in the mushroom cap interest area and the average whiteness value, and reflect whether the color of each mushroom cap is uniform and whether there are spots according to the deviation.

8. The method for digital characterization of Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 1, characterized in that: In step 4, the BP neural network of the genetic algorithm is used to classify the Agaricus bisporus.

9. The method for digital characterization of Agaricus bisporus phenotypic parameters based on an instance segmentation model according to claim 1, characterized in that: The grading standard is a three-level standard, among which the first level standard is that the color of the Agaricus bisporus is white and has no color spots, the cap shape is round or nearly round, and the cap diameter is greater than or equal to 5 cm; the second level standard is that the color of the Agaricus bisporus is white and has few color spots, the cap shape is round or nearly round, but there is slight deformity, and the cap diameter is 3.5-5 cm; the third level standard is that the color of the Agaricus bisporus is milky white with color spots, the cap shape is round or nearly round, but there is deformity, and the cap diameter is less than 3.5 cm.

Images