Plant segmentation method, device and equipment based on depth information
Through the plant segmentation method based on depth information, using depth estimation and semantic segmentation models, the pseudomask is automatically generated and the model is trained, which solves the problem of time-consuming and labor-consuming manual labeling in the existing technology, and improves the accuracy of plant image segmentation and efficiency.
Patent Information
- Application Number
- CN202411285683.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-13
AI Technical Summary
Existing plant segmentation models rely on manual data labeling, which makes the labeling process time-consuming and labor-intensive and error-free, and cannot meet the needs of high-throughput phenotypic analysis.
Using the plant segmentation method based on depth information, the monocular depth estimation model is used to generate a depth map, and the region mask transformation and segmentation threshold are determined, and the pseudo-mask is automatically generated, and the semantic segmentation model is trained through the pseudo-mask, which is finally improved by filtering. The accuracy of segmentation results.
Reliance on manual annotation is reduced, the determination accuracy and efficiency of pseudomasks are improved, the accuracy and efficiency of plant image segmentation are improved, and the segmentation efficiency can be significantly improved without affecting the segmentation accuracy.
Smart Images

Figure CN118799343B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a plant segmentation method, device and equipment based on depth information. Background Art
[0002] Currently, plant segmentation is an important part of plant analysis. After segmentation, the interference of other backgrounds outside the plants can be removed, and the plant phenotype can be analyzed more accurately.
[0003] However, some current plant segmentation models require corresponding training samples to rely on manually labeled data, which makes the labeling process time-consuming and labor-intensive, and contains errors, and cannot meet the needs of high-throughput phenotyping analysis. Summary of the invention
[0004] In view of this, the purpose of the present application is to propose a plant segmentation method, device and equipment based on depth information.
[0005] Based on the above objectives, the present application provides a plant segmentation method based on depth information, including:
[0006] Processing at least one acquired plant image using a monocular depth estimation model, and outputting a depth map corresponding to each plant image;
[0007] Performing a regional mask transformation process on the depth map to obtain a mask transformation result;
[0008] Determine a segmentation threshold, and use the segmentation threshold to perform segmentation processing on the mask transformation result to obtain a pseudo mask, wherein each plant image corresponds to a corresponding pseudo mask;
[0009] The corresponding plant images are marked using pseudo masks as sample sets;
[0010] The pre-built initial semantic segmentation model is trained using the sample set, and the parameters of the initial semantic segmentation model are adjusted according to the deviation between the training result and the corresponding pseudo mask to obtain a trained semantic segmentation model;
[0011] The plant image to be segmented is processed using a monocular depth estimation model to obtain a depth map to be segmented;
[0012] The trained semantic segmentation model is used to perform segmentation processing on the depth map to be segmented to obtain a segmentation mask, and the segmentation mask is filtered using the depth map to be segmented to obtain the final segmentation result.
[0013] Based on the same inventive concept, the present disclosure also provides a plant segmentation device based on depth information, comprising:
[0014] A depth estimation processing module is configured to process at least one acquired plant image using a monocular depth estimation model and output a depth map corresponding to each plant image;
[0015] A mask transformation processing module is configured to perform a regional mask transformation process on the depth map to obtain a mask transformation result;
[0016] A segmentation processing module is configured to determine a segmentation threshold, and use the segmentation threshold to perform segmentation processing on the mask transformation result to obtain a pseudo mask, wherein each plant image corresponds to a corresponding pseudo mask;
[0017] A sample determination module is configured to mark corresponding plant images using pseudo masks as a sample set;
[0018] A model training module is configured to use the sample set to train a pre-built initial semantic segmentation model, and adjust parameters of the initial semantic segmentation model according to a deviation between the training result and the corresponding pseudo mask to obtain a trained semantic segmentation model;
[0019] The depth estimation processing module is further configured to process the plant image to be segmented using a monocular depth estimation model to obtain a depth map to be segmented;
[0020] The model segmentation module is configured to use the trained semantic segmentation model to perform segmentation processing on the depth map to be segmented to obtain a segmentation mask, and use the depth map to be segmented to perform filtering processing on the segmentation mask to obtain a final segmentation result.
[0021] Based on the same inventive concept, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.
[0022] Based on the same inventive concept, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method as described above.
[0023] From the above, it can be seen that the plant segmentation method, device and equipment based on depth information provided by the present application can perform mask change processing on the depth map corresponding to the plant image, and can automatically perform segmentation processing according to the segmentation threshold to obtain a pseudo-mask corresponding to each plant image. This process of obtaining the pseudo-mask does not require manual processing, and can improve the accuracy and efficiency of determining the pseudo-mask; then, the pseudo-mask is automatically used to mark the plant image to form a sample, and the initial semantic segmentation model is trained to obtain a trained semantic segmentation model that can segment the plant image; finally, the monocular depth estimation model is used to process the plant image to be segmented to obtain a depth map to be segmented, and then the trained semantic segmentation model is used to segment the depth map to be segmented. In this way, the segmentation mask obtained by the deep algorithm segmentation technology (DepthCropSeg) is more accurate. In order to further improve the accuracy of the segmentation mask, the segmentation mask is filtered using the depth map to be segmented, so that the final segmentation result obtained is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 This is a flow chart of a plant segmentation method based on depth information according to an embodiment of the present application;
[0026] Figure 2 This is an architecture diagram of a plant segmentation process based on depth information according to an embodiment of the present application;
[0027] Figure 3 A schematic diagram of a process of generating a mask based on depth information according to an embodiment of the present application;
[0028] Figure 4 A schematic diagram of generating a pseudo mask in different scenarios of an embodiment of the present application;
[0029] Figure 5 A schematic diagram of a gradient-weighted grayscale transformation sequence and its fitting curve according to an embodiment of the present application;
[0030] Figure 6 This is a comparison diagram of the effects of the post-processing process based on depth information in an embodiment of the present application;
[0031] Figure 7 This is a schematic diagram of the structure of a plant segmentation device based on depth information according to an embodiment of the present application;
[0032] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0034] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0035] Plant segmentation can be divided into: herbaceous plant segmentation, woody plant segmentation, and can also be divided into agricultural crop segmentation and non-agricultural crop segmentation.
[0036] The following is a specific analysis of the problems in agricultural crop segmentation:
[0037] Crop segmentation is a key task in agricultural visual analysis and is widely used in crop growth monitoring, pest and disease assessment, and yield prediction, providing data support for agricultural decision-making. Crop segmentation results can extract phenotypic parameters such as canopy shape, coverage, and green projection area, thereby guiding agricultural production activities. General crop analysis is based on the transformation of image-based crop segmentation methods from the classic color-based paradigm to the recent deep learning-based paradigm. Deep learning-based crop segmentation methods often rely on a large amount of high-quality pixel-level manually annotated data, which makes the annotation process time-consuming and labor-intensive, and there are errors, which cannot meet the needs of high-throughput phenotypic analysis.
[0038] In order to reduce the reliance on manual labeling, a variety of alternative methods have been proposed in recent years, including weakly supervised learning, self-supervised learning, transfer learning, and semi-automatic labeling tools.
[0039] Weakly supervised learning reduces the need for precise pixel-level annotation by utilizing partially labeled data or weaker labels (such as image-level labels or bounding box labels) combined with unlabeled data for training. Some studies have proposed a method for crop organ segmentation and disease recognition based on weakly supervised deep convolutional neural networks (DCNNs), which only requires bounding boxes as labels. Self-supervised learning is to learn features from unlabeled data and then apply these features to crop segmentation tasks, further reducing the reliance on manual annotation. Some researchers have proposed a deep neural network (DNN) method that automatically identifies and prioritizes the most important genotypes and environmental variables through guided back-propagation feature selection techniques, thereby achieving accurate crop segmentation with reduced manual annotation. Transfer learning reduces the need for new annotations by applying a model that has been trained in one task or field to another similar task or field. For example, a two-step transfer learning method using a synthetic in vitro soybean pod dataset achieves migration from simulation to reality, and from in vitro segmentation to branch segmentation. Some semi-automatic crop annotation semi-automatic annotation methods based on deep learning algorithms are gradually emerging. More and more researchers have adopted the Segment Anything Model (SAM) to achieve semi-automatic labeling without further training. SAM has been trained with more than a billion images and has demonstrated high-quality, category-independent segmentation performance comparable to fully supervised models. Many researchers have begun to adopt LabelMe, a semi-automatic labeling tool based on SAM, to reduce the labeling cost in plant detection and segmentation tasks such as corn.
[0040] Although these methods have made some progress in reducing the reliance on manual annotation, there are still very limited studies comparing the performance of these methods with manually annotated data. In addition, these methods have limited savings in time cost and computing resources, further weakening their practicality. Therefore, it is urgent to develop an innovative technology that can not only reduce the need for manual annotation, but also significantly improve the segmentation efficiency without affecting the accuracy of crop segmentation, so as to better meet the needs of fast and efficient data processing in actual agricultural production.
[0041] Based on the above description, the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0042] The plant segmentation method based on depth information proposed in the embodiment of the present application is as follows: Figure 1 As shown, including:
[0043] Step 101: acquire at least one plant image (e.g., a crop image, such as Figure 2 As shown), a monocular depth estimation model is used for processing to output a depth map (e.g., a depth image) corresponding to each plant image.
[0044] In the specific implementation, the monocular depth estimation model selects the ViT-Large architecture model of Depth Anything Model (DAM) V2.
[0045] Use a camera to collect at least one plant image (with specifications of h×w×m, where h is the image height, w is the image width, and m is the number of feature channels, and it belongs to RGB image) of one or more plants. Input these plant images into the monocular depth estimation model respectively, and the depth map corresponding to each plant image (for example, a single-channel depth map) is obtained.
[0046] Step 102: Perform regional mask transformation processing on the depth map to obtain a mask transformation result.
[0047] In specific implementation, the corresponding depth map does not meet the conditions for segmentation, and it is necessary to perform regional mask transformation on the depth map, so as to obtain a mask transformation result that is easy to segment.
[0048] In some embodiments, step 102 includes:
[0049] Step 1021, normalize the depth map (for example, depth normalization, such as Figure 3 In step 1 shown in the figure), a normalized depth map is obtained.
[0050] In specific implementation, when normalizing the depth map, the minimum value of the corresponding row in the depth map is first subtracted from the row, and then the minimum value of the corresponding column in the depth map is subtracted from the column. This makes the background depth in the normalized depth map basically consistent, reducing the impact of different shooting angles.
[0051] Step 1022, performing gradient weighted histogram statistical processing on the normalized depth map to obtain a grayscale transformation sequence and a mask transformation result.
[0052] Through the above scheme, an accurate grayscale transformation sequence and mask transformation result can be obtained, so that the pseudo mask generated subsequently is more accurate.
[0053] In some embodiments, step 1022 includes:
[0054] Step 10221, determine the horizontal gradient value and the vertical gradient value of the normalized depth map, calculate the square sum of the horizontal gradient value and the vertical gradient value, and determine the square root based on the square sum to obtain the gradient map.
[0055] In specific implementation, the normalized depth map is ( Represents the pixel value of the i-th row and j-th column in the normalized depth map) Use the Sobel operator to calculate the horizontal gradient and the vertical gradient , take the L2 norm of the two directional gradients as the final gradient map .in Represents the pixel value of the i-th row and j-th column in the gradient image.
[0056] Step 10222, separate and discretize the gradient map to obtain a grade distribution gradient matrix.
[0057] In specific implementation, the number of gradient levels is determined to be p, and the corresponding level distribution gradient matrix L is obtained, and the formula is:
[0058] ;
[0059] in, l ij Indicates the level of the pixel in the i-th row and j-th column, and its value range is an integer between 0 and p (excluding p). represents the maximum gradient value in the gradient map obtained by the above calculation, represents the value of the i-th row and j-th column in the gradient image. The calculation process is to normalize the gradient image to the range of 0 to p and then round it down.
[0060] Step 10223: Perform edge detection on the normalized depth map to obtain an edge map.
[0061] In specific implementation, the Canny operator is used to detect the edge map on the normalized depth map (such as Figure 3 As shown, the corresponding step 2) ,in Represents the value of the i-th row and j-th column in the edge graph E, where edge points are 1 and non-edge points are 0.
[0062] In some embodiments, step 10223 further includes:
[0063] Step 102231, the edge map is subjected to morphological dilation processing to connect partially disconnected lines in the edge map to obtain a processed edge map.
[0064] Step 102232, using a boundary tracing algorithm to find the contour in the processed edge map, and obtain the corresponding region mask.
[0065] The rough outline of the area of interest is obtained by using the boundary tracking method, and the corresponding area mask F is generated, where the inside of the area of interest is 1 and the rest is 0.
[0066] Step 10224, determine the gradient level weight based on the level distribution gradient matrix, and construct a grayscale transformation sequence according to the gradient level weight.
[0067] In specific implementation, the gradient weight of each gradient level is the gradient of the corresponding level pixel and the proportion of the total gradient, and the calculation formula is: .in represents the weight of the k-th level gradient, l ij represents the gradient level corresponding to the pixel in the i-th row and j-th column, Represents the value of the i-th row and j-th column in the edge graph E. Constructing a grayscale transformation sequence , where n represents a grayscale value between 0 and 255.
[0068] Step 10225: Use the grayscale transformation sequence to perform transformation processing on the normalized depth map to obtain a grayscale transformation result.
[0069] In specific implementation, the normalized grayscale transformation sequence is: .use The normalized depth map is transformed to obtain a mask as the grayscale transformation result.
[0070] Step 10226, performing regional mask transformation processing on the grayscale transformation result to obtain a mask transformation result.
[0071] In specific implementation, the grayscale transformation result and the above generated region of interest mask F are intersected to obtain the final mask transformation result W.
[0072] Through the above solution, it can be ensured that the mask change result is more accurate.
[0073] Step 103, determining a segmentation threshold, and using the segmentation threshold to perform segmentation processing on the mask transformation result to obtain a pseudo mask, wherein each plant image corresponds to a corresponding pseudo mask.
[0074] In specific implementation, the corresponding segmentation threshold can be set according to actual needs, or it can be a segmentation threshold obtained by automatic calculation. The mask transformation result is segmented according to the segmentation threshold. The part less than the segmentation threshold belongs to the background part, and the part greater than the segmentation threshold belongs to the foreground part. For example, the background part in the mask transformation result is set to "0" and the foreground part is set to "1".
[0075] This process of determining the pseudo-mask is completed automatically and does not require human intervention, which can improve the efficiency of pseudo-mask determination (such as Figure 4 As shown in Figure 2, the depth maps and pseudo masks corresponding to various crops are obtained).
[0076] In some embodiments, determining the segmentation threshold in step 103 includes:
[0077] Step 1031, construct a logistic regression parameter equation, and use an iterative optimization algorithm to fit the grayscale transformation sequence to obtain a fitting function (such as Figure 3 As shown, the corresponding step is step three).
[0078] When it comes to specific implementation, Among them, x is the variable of the equation, and a, b, and c are the parameters to be fitted in the equation. Since the grayscale value range is between 0 and 255, the parameter c in the formula can be directly fixed to 255, the value range of parameter b is specified to be between 0 and 255, and the value range of parameter a is selected to be between 0 and 5. The above parametric equation is fitted using the Levenberg-Marquardt algorithm (sigmoid) , and get the fitting equation ,like Figure 3 (correspond Figure 3 The curve fit in the graph is presented in Figure 5 ( Figure 5 Yes Figure 3 ) is shown in the enlarged view of the curve graph in FIG.
[0079] Step 1032: Select the point with the largest slope in the fitting function as the segmentation threshold.
[0080] In specific implementation, select the fitting equation When the x value is between 0 and 255, the horizontal coordinate of the point with the maximum slope is the segmentation threshold. ,like Figure 3 and Figure 5 Middle red dashed line.
[0081] The mask transformation result W generated above is subjected to threshold segmentation to obtain the final pseudo mask M. Specifically, the pixel value in W is greater than the segmentation threshold. The part smaller than the segmentation threshold is set to 1, indicating the foreground, and the part smaller than the segmentation threshold is set to 0, indicating the background.
[0082] Step 104: Use a pseudo mask to mark the corresponding plant images as a sample set.
[0083] In a specific implementation, the pseudo mask is marked on the plant image as a label of the plant image, and no manual marking is required, which can save human resources and improve the efficiency of marking.
[0084] The sample set includes: a training set, and the training set includes: a plurality of training samples, each training sample is a plant image marked with a pseudo mask.
[0085] In some embodiments, the sample set further includes: a test set, wherein the test set includes: a plurality of test samples, each test sample being a plant image marked with a pseudo mask.
[0086] Step 105 , using the sample set to train the pre-built initial semantic segmentation model, adjusting the parameters of the initial semantic segmentation model according to the deviation between the training result and the corresponding pseudo mask, to obtain a trained semantic segmentation model.
[0087] In specific implementation, the pre-built initial semantic segmentation model can be a SegFormer model (the backbone network of which is MixVisionTransformer-B5).
[0088] In some embodiments, step 105 includes:
[0089] Step 1051, using the training set to train a pre-built initial semantic segmentation model (eg, SegFormer) through a gradient back propagation algorithm, wherein each training sample in the training process outputs a training output result.
[0090] Step 1052, calculate the evaluation index IoU corresponding to each training sample according to the training output result M' and the marked pseudo mask, and count the average value mIoU of the evaluation index of each training sample, determine the loss function according to the average value mIoU, and adjust the parameters of the initial semantic segmentation model to obtain a trained semantic segmentation model.
[0091] Among them, IoU = (M'∩M) / (M'∪M).
[0092] In some embodiments, after step 1052, the method further includes:
[0093] Step 1053: Use the trained semantic segmentation model to perform inference prediction on each test sample in the test set to obtain the prediction result of the test sample. .
[0094] Step 1054: The prediction result of the test sample Combined with the pseudo mask M corresponding to the test sample, a trisection map is generated .
[0095] The specific formula for the three-point diagram is:
[0096] ;
[0097] Among them, i represents the horizontal coordinate, j represents the vertical coordinate, Represents an element-wise exclusive-or operation.
[0098] Step 1055: using the three-part image The trained semantic segmentation model (e.g., SegFormer) is trained again using the training labels, and the loss function of the re-training is calculated. The parameters in the trained semantic segmentation model are adjusted according to the loss function to obtain the final trained semantic segmentation model; wherein, the part corresponding to the highest grayscale value (e.g., the grayscale value is 255) in the training label does not participate in the loss function calculation.
[0099] Step 106: Process the plant image to be segmented using a monocular depth estimation model to obtain a depth map to be segmented.
[0100] In specific implementation, after the semantic segmentation model is trained, the trained semantic segmentation model can be used for segmentation processing. However, the trained semantic segmentation model cannot directly process the plant image to be segmented, and a monocular depth estimation model is required to perform depth processing to obtain a depth map D to be segmented.
[0101] Step 107, using the trained semantic segmentation model to perform segmentation processing on the depth map to be segmented to obtain a segmentation mask, and using the depth map to be segmented to perform filtering processing on the segmentation mask to obtain a final segmentation result (such as Figure 6 As shown, it corresponds to the processing result based on the depth information).
[0102] In the specific implementation, the depth map to be segmented is input into the trained semantic segmentation model for segmentation processing. However, there may be some noise in the segmentation process, and the obtained segmentation mask is not accurate enough. It is necessary to use the depth map D to be segmented obtained in step 106 to filter the segmentation mask (for example, joint bilateral filtering), so that the final segmentation result is obtained. It is more accurate.
[0103] Final segmentation result for .
[0104] Through the above scheme, the depth map corresponding to the plant image can be processed by mask change, and the segmentation processing can be automatically performed according to the segmentation threshold to obtain the pseudo-mask corresponding to each plant image. This process of obtaining the pseudo-mask does not require manual processing, which can improve the accuracy and efficiency of determining the pseudo-mask; then, the pseudo-mask is automatically used to mark the plant image to form a sample, and the initial semantic segmentation model is trained to obtain a trained semantic segmentation model that can segment the plant image; finally, the monocular depth estimation model is used to process the plant image to be segmented to obtain the depth map to be segmented, and then the trained semantic segmentation model is used to segment the depth map to be segmented, so that the segmentation mask obtained by the deep algorithm segmentation technology (DepthCropSeg) is more accurate. In order to further improve the accuracy of the segmentation mask, the segmentation mask is filtered using the depth map to be segmented, so that the final segmentation result is more accurate.
[0105] In summary:
[0106] (1) The solution of this application does not require manual annotation work at the pixel level. Instead, it uses a threshold segmentation method based on a monocular depth estimation model to generate a pseudo mask for plant images, thereby achieving performance comparable to that of using manually annotated data while greatly improving the efficiency of the crop segmentation task.
[0107] (2) The solution of the present application adopts a self-training method to train the initial semantic segmentation model. By using the intersection of the inference result of the trained semantic segmentation model on the training set and the pseudo-mask as the label for the next training, the performance of the semantic segmentation model is improved to a certain extent.
[0108] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.
[0109] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0110] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a plant segmentation device based on depth information.
[0111] refer to Figure 7 , the device comprises:
[0112] The depth estimation processing module 201 is configured to process the acquired at least one plant image using a monocular depth estimation model and output a depth map corresponding to each plant image;
[0113] The mask transformation processing module 202 is configured to perform a regional mask transformation process on the depth map to obtain a mask transformation result;
[0114] A segmentation processing module is configured to determine a segmentation threshold, and use the segmentation threshold to perform segmentation processing on the mask transformation result to obtain a pseudo mask, wherein each plant image corresponds to a corresponding pseudo mask;
[0115] The sample determination module 203 is configured to mark the corresponding plant images using pseudo masks as sample sets;
[0116] The model training module 204 is configured to use the sample set to train the pre-built initial semantic segmentation model, and adjust the parameters of the initial semantic segmentation model according to the deviation between the training result and the corresponding pseudo mask to obtain a trained semantic segmentation model;
[0117] The depth estimation processing module 201 is further configured to process the plant image to be segmented using a monocular depth estimation model to obtain a depth map to be segmented;
[0118] The model segmentation module 205 is configured to use the trained semantic segmentation model to perform segmentation processing on the depth map to be segmented to obtain a segmentation mask, and use the depth map to be segmented to perform filtering processing on the segmentation mask to obtain a final segmentation result.
[0119] In some embodiments, the mask transformation processing module 202 includes:
[0120] A normalization processing unit, configured to perform normalization processing on the depth map to obtain a normalized depth map;
[0121] The weighted change processing is configured to perform gradient weighted histogram statistical processing on the normalized depth map to obtain a grayscale transformation sequence and a mask transformation result.
[0122] In some embodiments, the weighted change processing is specifically configured as follows:
[0123] Determine the gradient value in the horizontal direction and the gradient value in the vertical direction of the normalized depth map, calculate the sum of the squares of the gradient value in the horizontal direction and the gradient value in the vertical direction, and determine the square root based on the square sum to obtain the gradient map;
[0124] Separating and discretizing the gradient map to obtain a grade distribution gradient matrix;
[0125] Perform edge detection on the normalized depth map to obtain an edge map;
[0126] Determine a gradient level weight based on the level distribution gradient matrix, and construct a grayscale transformation sequence according to the gradient level weight and the edge map;
[0127] Using the grayscale transformation sequence to perform a transformation process on the normalized depth map to obtain a grayscale transformation result;
[0128] Performing regional mask transformation processing on the grayscale transformation result to obtain a mask transformation result.
[0129] In some embodiments, the weighted change processing is further configured as follows:
[0130] After edge detection is performed on the normalized depth map to obtain an edge map:
[0131] Connecting disconnected lines in the edge map by morphological dilation processing on the edge map to obtain a processed edge map;
[0132] Use the boundary tracking algorithm to find the contours in the processed edge map and obtain the corresponding region mask;
[0133] The performing a regional mask transformation process on the grayscale transformation result to obtain a mask transformation result includes:
[0134] The grayscale transformation result is subjected to intersection processing with the region mask to obtain a mask transformation result.
[0135] In some embodiments, the segmentation processing module is specifically configured to:
[0136] Constructing a logistic regression parameter equation, and fitting the grayscale transformation sequence to the logistic regression parameter equation using an iterative optimization algorithm to obtain a fitting function;
[0137] The point with the largest slope in the fitting function is selected as the segmentation threshold.
[0138] In some embodiments, the sample set includes: a training set, the training set includes: a plurality of training samples, each training sample is a plant image marked with a pseudo mask;
[0139] The model training module 204 is specifically configured to:
[0140] The pre-built initial semantic segmentation model is trained using the training set through the gradient back-propagation algorithm. During the training process, each training sample outputs a training output result.
[0141] The evaluation index corresponding to each training sample is calculated according to the training output result and the marked pseudo mask, and the average value of the evaluation index of each training sample is statistically calculated. The loss function is determined according to the average value to adjust the parameters of the initial semantic segmentation model to obtain a trained semantic segmentation model.
[0142] In some embodiments, the sample set further includes: a test set, wherein the test set includes: a plurality of test samples, each test sample being a plant image marked with a pseudo mask.
[0143] The model training module 204 is configured to:
[0144] After obtaining the trained semantic segmentation model, use the trained semantic segmentation model to perform inference prediction on each test sample in the test set to obtain a prediction result of the test sample;
[0145] Combining the prediction result of the test sample with the pseudo mask corresponding to the test sample to generate a trisection map;
[0146] The trained semantic segmentation model is trained again using the triplicate map as a training label, and the loss function of the re-training is calculated. The parameters in the trained semantic segmentation model are adjusted according to the loss function to obtain the final trained semantic segmentation model; wherein the part corresponding to the highest grayscale value in the training label does not participate in the loss function calculation.
[0147] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0148] The device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0149] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the above embodiments when executing the program.
[0150] Figure 8A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.
[0151] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0152] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0153] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0154] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0155] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).
[0156] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0157] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0158] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.
[0159] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0160] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0161] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.
[0162] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.
[0163] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0164] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0165] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0166] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0167] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device may be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0168] Although the present application has been described in conjunction with specific embodiments of the present application, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the discussed embodiments.
[0169] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.
Claims
1. A plant segmentation method based on depth information, characterized in that: include: Processing at least one acquired plant image using a monocular depth estimation model, and outputting a depth map corresponding to each plant image; Performing a regional mask transformation process on the depth map to obtain a mask transformation result; Determine a segmentation threshold, and use the segmentation threshold to perform segmentation processing on the mask transformation result to obtain a pseudo mask, wherein each plant image corresponds to a corresponding pseudo mask; The corresponding plant images are marked using pseudo masks as sample sets; The pre-built initial semantic segmentation model is trained using the sample set, and the parameters of the initial semantic segmentation model are adjusted according to the deviation between the training result and the corresponding pseudo mask to obtain a trained semantic segmentation model; The plant image to be segmented is processed using a monocular depth estimation model to obtain a depth map to be segmented; The trained semantic segmentation model is used to perform segmentation processing on the depth map to be segmented to obtain a segmentation mask, and the segmentation mask is filtered using the depth map to be segmented to obtain the final segmentation result.
2. The method according to claim 1, characterized in that The performing a regional mask transformation process on the depth map to obtain a mask transformation result includes: Normalizing the depth map to obtain a normalized depth map; The normalized depth map is statistically processed by gradient weighted histogram to obtain the grayscale transformation sequence and mask transformation results.
3. The method according to claim 2, characterized in that The normalized depth map is subjected to gradient weighted histogram statistical processing to obtain a grayscale transformation sequence and a mask transformation result, including: Determine the gradient value in the horizontal direction and the gradient value in the vertical direction of the normalized depth map, calculate the sum of the squares of the gradient value in the horizontal direction and the gradient value in the vertical direction, and determine the square root based on the square sum to obtain a gradient map; Separating and discretizing the gradient map to obtain a grade distribution gradient matrix; Perform edge detection on the normalized depth map to obtain an edge map; Determine a gradient level weight based on the level distribution gradient matrix, and construct a grayscale transformation sequence according to the gradient level weight and the edge map; Using the grayscale transformation sequence to perform a transformation process on the normalized depth map to obtain a grayscale transformation result; Performing regional mask transformation processing on the grayscale transformation result to obtain a mask transformation result.
4. The method according to claim 3, characterized in that After edge detection is performed on the normalized depth map to obtain an edge map, the method further includes: Connecting disconnected lines in the edge map by morphological dilation processing on the edge map to obtain a processed edge map; Use the boundary tracking algorithm to find the contours in the processed edge map and obtain the corresponding region mask; The performing a regional mask transformation process on the grayscale transformation result to obtain a mask transformation result includes: The grayscale transformation result is subjected to intersection processing with the region mask to obtain a mask transformation result.
5. The method according to claim 3, characterized in that: The step of determining the segmentation threshold comprises: Constructing a logistic regression parameter equation, and fitting the grayscale transformation sequence to the logistic regression parameter equation using an iterative optimization algorithm to obtain a fitting function; The point with the largest slope in the fitting function is selected as the segmentation threshold.
6. The method according to claim 1, characterized in that The sample set includes: a training set, wherein the training set includes: a plurality of training samples, each training sample being a plant image marked with a pseudo mask; The method of using the sample set to train the pre-built initial semantic segmentation model, adjusting the parameters of the initial semantic segmentation model according to the deviation between the training result and the corresponding pseudo mask, and obtaining the trained semantic segmentation model includes: The pre-built initial semantic segmentation model is trained using the training set through the gradient back-propagation algorithm. During the training process, each training sample outputs a training output result. The evaluation index corresponding to each training sample is calculated according to the training output result and the marked pseudo mask, and the average value of the evaluation index of each training sample is statistically calculated. The loss function is determined according to the average value to adjust the parameters of the initial semantic segmentation model to obtain a trained semantic segmentation model.
7. The method according to claim 6, characterized in that The sample set also includes: a test set, the test set includes: a plurality of test samples, each test sample is a plant image marked with a pseudo mask; After obtaining the trained semantic segmentation model, it also includes: Using the trained semantic segmentation model to perform inference prediction on each test sample in the test set to obtain a prediction result for the test sample; Combining the prediction result of the test sample with the pseudo mask corresponding to the test sample to generate a trisection map; The trained semantic segmentation model is trained again using the triplicate map as a training label, and the loss function of the re-training is calculated. The parameters in the trained semantic segmentation model are adjusted according to the loss function to obtain the final trained semantic segmentation model; wherein the part corresponding to the highest grayscale value in the training label does not participate in the loss function calculation.
8. A plant segmentation device based on depth information, characterized in that: include: A depth estimation processing module is configured to process at least one acquired plant image using a monocular depth estimation model and output a depth map corresponding to each plant image; A mask transformation processing module is configured to perform a regional mask transformation process on the depth map to obtain a mask transformation result; A segmentation processing module is configured to determine a segmentation threshold, and use the segmentation threshold to perform segmentation processing on the mask transformation result to obtain a pseudo mask, wherein each plant image corresponds to a corresponding pseudo mask; A sample determination module is configured to mark corresponding plant images using pseudo masks as a sample set; A model training module is configured to use the sample set to train a pre-built initial semantic segmentation model, and adjust parameters of the initial semantic segmentation model according to a deviation between the training result and the corresponding pseudo mask to obtain a trained semantic segmentation model; The depth estimation processing module is further configured to process the plant image to be segmented using a monocular depth estimation model to obtain a depth map to be segmented; The model segmentation module is configured to use the trained semantic segmentation model to perform segmentation processing on the depth map to be segmented to obtain a segmentation mask, and use the depth map to be segmented to perform filtering processing on the segmentation mask to obtain a final segmentation result.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.
