Single plant segmentation method, device, terminal device and readable storage medium

Abstract

The present application provides a method, apparatus, terminal device, and readable storage medium for segmenting a single plant. The method comprises obtaining an image of a plant to be identified; wherein the image of the plant to be identified includes one or more plants; then inputting the image of the plant to be identified into a pre-trained plant segmentation model to output a segmentation result of the single plant; wherein the plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples, and the improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROI Align network, and a fully connected layer connected in sequence. This method can achieve segmentation of small plants with high efficiency and accuracy.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, terminal device, and computer-readable storage medium for segmenting a single plant. Background Art

[0002] Vegetation is an important cover of the earth's surface. Vegetation leaves control the carbon cycle and energy exchange in the ecosystem through photosynthesis and transpiration, thereby controlling the balance between vegetation productivity and natural systems. At present, the research on vegetation and atmospheric changes is based on the exchange of substances between the vegetation canopy and the atmosphere. The research on the interaction between individual plants and the atmosphere is affected by the limitations of the acquisition methods. Based on the development of remote sensing satellites and near-ground remote sensing detection methods, vegetation observation research has also been refined from the vegetation canopy to the scale of individual trees. Single tree segmentation refers to the segmentation of individual plants from remote sensing images. The forest structure parameters of individual plants can be used for subsequent research on the exchange between vegetation and the atmosphere.

[0003] Traditionally, individual tree extraction involves field forest resource surveys to obtain plant information such as tree height, diameter at breast height, and crown width. These traditional methods are difficult to measure, require long construction times, and have low information collection efficiency. With the development of remote sensing satellites and near-ground satellites, vegetation data can now be recorded using ground-based or airborne lidar, offering new research opportunities for individual tree extraction. Ground-based lidar uses a bottom-up recording method, which limits the height at which data can be acquired and results in blind spots when measuring the canopy layer. Airborne lidar uses a top-down recording method, which provides a better perspective for recording the vertical structural features of vegetation. Laser point cloud data acquired by lidar is commonly used to extract individual trees. Extraction methods typically construct a series of models to invert vegetation characteristics and perform individual tree segmentation. Methods for individual tree segmentation using laser point cloud data include local maximum methods, watershed-based methods, point cloud clustering methods, PCS algorithms, polynomial fitting based on CHM, and mean shift with region merging. These methods all process the texture characteristics of vegetation in image data and recorded laser point cloud data, using various fitting methods to extract parameters such as tree height and crown diameter to determine the segmentation of individual plants. The accuracy of individual tree segmentation is often affected by the volume of laser point cloud data detected. The point cloud data is too sparse to depict the vertical distribution of the vegetation canopy, thus failing to guarantee accurate individual tree segmentation.

[0004] For trees with simple structures and sparse distribution, some researchers have proposed a point cloud data processing method that combines supervoxels for tree segmentation. This method uses mobile laser scanning data, analyzes geometric features using supervoxel clustering, and then labels trees using geometric context features and a random forest method. Iterative regularization of local information is then used to optimize global information, spatially smoothing the initial semantic classification results. Finally, individual trees are segmented based on the semantic segmentation labeling results. This method effectively preserves the geometric boundaries of individual trees and improves the efficiency of individual tree segmentation. However, for trees with complex and narrow structures, over-segmentation and under-segmentation can still occur.

[0005] With the development of deep learning, some deep learning methods have also been used to segment tree images from drone imagery. For example, the U-net convolutional neural network is used to segment vegetation areas in high-resolution imagery. However, due to its pixel-level detection, it can only be used to segment large shrubs. A deep learning-based oil palm tree detection and counting method uses simple deep learning models as classifiers to segment individual trees at a small regional scale. The deep learning method for segmenting individual plants directly segments vegetation areas from images, which is more convenient and faster than traditional inversion methods. However, it can currently only segment large shrubs and cannot accurately segment small plants. Summary of the Invention

[0006] In view of this, the embodiments of the present application provide a method, apparatus, terminal device and computer-readable storage medium for segmenting a single plant to overcome the problem that the deep learning method for segmenting a single plant in the prior art can only be used to segment large shrubs and cannot accurately segment small plants.

[0007] In a first aspect, an embodiment of the present application provides a method for segmenting a single plant, the method comprising:

[0008] Acquire a plant image to be identified; wherein the plant image to be identified includes one or more plants;

[0009] The plant image to be identified is input into a pre-trained plant segmentation model to output the segmentation result of a single plant; wherein the plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples, and the improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROI Align network and a fully connected layer connected in sequence.

[0010] In a second aspect, an embodiment of the present application provides a device for dividing a single plant, the device comprising:

[0011] A module for acquiring an image of a plant to be identified, configured to acquire an image of a plant to be identified; wherein the image of the plant to be identified includes one or more plants;

[0012] A segmentation result module is used to input the plant image to be identified into a pre-trained plant segmentation model to output the segmentation result of a single plant; wherein the plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples, and the improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROI Align network and a fully connected layer connected in sequence.

[0013] In a third aspect, an embodiment of the present application provides a terminal device comprising: a memory; one or more processors coupled to the memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the single plant segmentation method provided in the first aspect above.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method for dividing a single plant provided in the first aspect.

[0015] The embodiments of the present application provide a method, apparatus, terminal device, and computer-readable storage medium for segmenting a single plant. The method first obtains a plant image to be identified, wherein the plant image to be identified includes one or more plants. The plant image to be identified is then input into a pre-trained plant segmentation model to output a single plant segmentation result. The plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples. The improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROI Align network, and a fully connected layer connected in sequence.

[0016] The single plant segmentation method in the embodiment of the present application uses labeled plant image samples to train the improved CBAM-MaskRCNN model to obtain a pre-trained plant segmentation model. Based on the plant segmentation model, single plants in the plant image to be identified can be identified, thereby realizing the segmentation of single plants, which is very convenient to use; and the improved CBAM-MaskRCNN model is used to add the CBAM attention module and the RPN network to the traditional convolutional model, which can realize the segmentation of small plants with high segmentation efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0018] Figure 1 A schematic diagram of an application scenario of the method for segmenting a single plant provided in an embodiment of the present application;

[0019] Figure 2 A schematic flow chart of a method for segmenting a single plant provided in one embodiment of the present application;

[0020] Figure 3 A schematic diagram of the improved CBAM-MaskRCNN model structure provided in one embodiment of the present application;

[0021] Figure 4 This is a schematic structural diagram of a device for dividing a single plant provided in one embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of the structure of a terminal device provided in one embodiment of the present application;

[0023] Figure 6 A schematic diagram of the structure of a computer-readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The following is a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In order to explain the present application in more detail, the following specifically describes a method, apparatus, terminal device and computer-readable storage medium for segmenting a single plant provided by the present application in conjunction with the accompanying drawings.

[0026] Please refer to Figure 1 , Figure 1A schematic diagram illustrates an application scenario for the method for segmenting a single plant provided in an embodiment of the present application. This application scenario includes a terminal device 100 provided in an embodiment of the present application. Terminal device 100 may be any electronic device with a display screen (e.g., structures 102, 104, 106, and 108), including but not limited to a smartphone and a computer device, wherein the computer device may be at least one of a desktop computer, a portable computer, a laptop computer, a tablet computer, and the like. A pre-trained plant segmentation model may be stored in terminal device 100; terminal device 100 may also be used to take a photo of a plant to be identified and generate an image of the plant to be identified.

[0027] Terminal device 100 may generally refer to one of multiple terminal devices. This embodiment uses terminal device 100 as an example. Those skilled in the art will appreciate that the number of terminal devices may be greater or lesser. For example, there may be only a few terminal devices, or dozens, hundreds, or even more terminal devices. This embodiment of the present application does not limit the number and type of terminal devices. Terminal device 100 may be used to perform a method for segmenting a single plant provided in the embodiments of this application.

[0028] In an optional embodiment, the application scenario includes, in addition to the terminal device 100 provided in the embodiment of the present application, a server, wherein a network is provided between the server and the terminal device. The network is used as a medium for providing a communication link between the terminal device and the server. The network can include various connection types, such as wired or wireless communication links or fiber optic cables.

[0029] It should be understood that the number of terminal devices, networks, and servers is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers may be provided. For example, a server may be a server cluster consisting of multiple servers. The terminal device interacts with the server via the network to receive or send messages, etc. The server may be a server that provides various services. The server may be used to execute the steps of a method for segmenting a single plant provided in an embodiment of the present application. In addition, when executing a method for segmenting a single plant provided in an embodiment of the present application, the terminal device may execute part of the steps on the terminal device and part of the steps on the server, which is not limited here.

[0030] Based on this, a method for dividing a single plant is provided in the embodiment of the present application. Figure 2 , Figure 2 A schematic diagram of a process for segmenting a single plant provided in an embodiment of the present application is shown. Figure 1 The terminal device in the example is used to illustrate the process, including the following steps:

[0031] Step S110 , obtaining a plant image to be identified; wherein the plant image to be identified includes one or more plants.

[0032] Specifically, the plant image to be identified can be any image requiring individual plant identification or segmentation. The plant image to be identified may contain one or more plants. When there are multiple plants, the plants may be separated, i.e., the plants do not cluster or overlap with each other; or they may be clustered, i.e., the plants overlap with each other.

[0033] In addition, the angle (i.e., whether the image is tilted), color, size, resolution, etc. of the plant image to be identified are not limited, as long as they meet the minimum requirements for image recognition.

[0034] In step S120 , the plant image to be identified is input into a pre-trained plant segmentation model to output a single plant segmentation result.

[0035] Among them, the plant segmentation model is obtained by training the improved CBAM-MaskRCNN model using labeled plant image samples. The improved CBAM-MaskRCNN model includes a backbone network, CBAM module, RPN network, ROI Align network and fully connected layer connected in sequence.

[0036] Specifically, model training is to give an input vector and a target output value, then input the input vector into one or more network structures or functions to obtain the actual output value, calculate the bias based on the target output value and the actual output value, and judge whether the bias is within the allowable range; if it is within the allowable range, the training ends and the relevant parameters are fixed; if it is not within the allowable range, some parameters in the network structure or function are continuously adjusted until the bias is within the allowable range or a certain end condition is reached, the training ends and the relevant parameters are fixed, and finally the trained model can be obtained based on the fixed relevant parameters.

[0037] The training of the plant segmentation model in this embodiment is actually as follows: the marked plant image samples marked with plant information are input as input vectors to the improved CBAM-MaskRCNN model, and the area where the single plant is located is used as the target output value; the hidden layer is calculated, and the deviation between the target output value and the actual output value is calculated according to the output of each layer unit. When the deviation is within the unacceptable range, the error of the neurons in the network layer is calculated, the error gradient is calculated, and the weights are updated. The hidden layer is calculated again, and the deviation between the target output value and the actual output value is calculated according to the output of each layer unit until the deviation is within the acceptable range. The training is completed, and the weights and thresholds are fixed to obtain a pre-trained plant segmentation model. In addition, in this embodiment, an improved CBAM-MaskRCNN model is adopted, and the improvement includes a backbone network, a CBAM module, an RPN network, a ROI Align network and a fully connected layer connected in sequence (please refer to Figure 3 The CBAM module is the attention module of the convolutional module, which is an addition to the traditional convolutional model. The introduction of the attention mechanism in the CBAM-MaskRCNN model can further improve the accuracy of single plant recognition or segmentation.

[0038] Furthermore, an implementation method for training or establishing a plant segmentation model is also provided, and the relevant description is as follows:

[0039] In one embodiment, a method for training a plant segmentation model includes:

[0040] S1: Obtain plant image samples, and use plant information to mark each plant in the plant image samples.

[0041] In this embodiment, a relatively large number (e.g., several thousand, tens of thousands, etc.) of plant image samples must first be prepared. The plant image samples can be collected by a photographing device itself, or can be downloaded from a database.

[0042] In a preferred embodiment, the plant image samples are derived from the PhenoCam data provided by ORNL DAAC, a subsidiary laboratory of NASA. The PhenoCam network (http: / / phenocam.sr.unh.edu) is a near-ground monitoring data. The network was established in 2008 and currently has more than 750 interannual phenological sites, mainly distributed in North America and Canada, from Alaska to Florida, from Hawaii to Arizona to Maine, and also in Central and South America and Europe. The PhenoCam phenological sites record vegetation phenological images at high temporal resolution using digital camera images, recording images containing red, green, and blue bands every 30 minutes from dawn to dusk. Infrared bands are also added to some local sites for detection, and the time frequency at night is relatively extended. The present invention selected phenological camera images from 46 site areas within a year, and each site selected one year of data as the research area, covering the latitude range of 32° to 47°N and the elevation range of 1-1550m. Most of these research sites have a temperate continental climate. Each site selected phenological camera image data between 11:30 noon and 13:30 pm local time for the experiment.

[0043] The plant image samples in the embodiments of the present application select PhenoCam data, firstly because the data is public data and easy to obtain. The phenological camera image data transmitted in real time by the site is publicly available on the website platform, and users can download it directly. Secondly, the data covers a wide range, including vegetation types in areas with different climate types. Currently, the data includes 1,783 site-years of data as of the end of 2018, spanning a range from North America to parts of South America. Third, the data and images of the site platform have been screened, and grayscale camera data using automatic white balance settings have been excluded. Since the automatic white balance setting will have a negative impact on the quality of the derived vegetation information data, in order to improve the accuracy of single plant segmentation, it is proposed in the embodiments of the present application that the white balance data is eliminated.

[0044] Generally speaking, the more plant image samples there are, the more accurate the trained model will be; however, too many plant image samples will slow down the model training process. Therefore, in practical applications, it is sufficient to select an appropriate number of image samples.

[0045] In this embodiment, when obtaining phenological camera data from the vegetation phenological camera PhenoCam data, the phenological camera data is also preprocessed. The preprocessing process starts with selecting site data, and priority is given to site data with continuous data and less loss in the public data set. Some selected site data are removed due to data pollution, including weather reasons such as rain wetting the lens, resulting in water droplets in the captured image, or noise interference caused by lens fog or even sun spots, or pollution such as animal footprints. Finally, considering factors such as climate distribution and vegetation types at different sites, a total of 7,639 phenological camera images of 46 site data from the website platform were selected for training and verification in the embodiment of this application. In addition, when preparing plant image samples, the samples are diversified as much as possible.

[0046] Furthermore, after obtaining the plant image samples, the plant image samples can be divided into a training set and a test set according to a preset ratio (e.g., 7:3). The plant image samples in the training set and the test set are then labeled with plant information. Optionally, a plant label can be used to label each plant in the plant image sample.

[0047] In an optional embodiment, labelimg software can be used to mark plant image samples, and each plant is selected on the plant image sample, where each label sample object is defined as a tree, and each tree is composed of a separate closed polygonal area, thereby generating plant labels, and saving the plant labels in a corresponding (e.g., xml) format file.

[0048] After labeling is completed, the labeled plant image samples in the training set can be used to train the improved CBAM-MaskRCNN model to obtain a plant segmentation model. After obtaining the plant segmentation model, the labeled plant image samples in the test set can be used to verify the accuracy of the plant segmentation model.

[0049] When using labeled plant image samples to train the improved CBAM-MaskRCNN model, some processing can be performed on the plant image samples before or after labeling, such as image enhancement and image size adjustment, to improve the generalization ability of the model and avoid overfitting.

[0050] Next, an example of processing plant image samples is given, which is described in detail as follows:

[0051] In one embodiment, before using plant information to mark each plant in the plant image sample, the method further includes: randomly selecting one or more enhancement processing methods to perform enhancement processing on the plant image sample; wherein when multiple enhancement processing methods are selected, the running order of each enhancement processing method is random, and the enhancement processing method includes grayscale transformation processing, random offset transformation processing, random folding processing, random scaling processing, random stretching processing and random rotation processing on the plant image sample.

[0052] Specifically, plant image sample enhancement plays a crucial role in the plant segmentation model's individual plant recognition or segmentation performance. Data augmentation also helps improve the generalization capabilities of the improved CBAM-MaskRCNN model. Unenhanced plant image samples are typically smaller in size than those that have undergone image enhancement. Directly using them in image recognition, classification, and segmentation model training often leads to overfitting. To train an excellent plant segmentation model, data augmentation is performed on the labeled plant image samples. Furthermore, to avoid overfitting due to small data volumes, the plant image samples need to be expanded.

[0053] In this embodiment, the following methods can be used to enhance the plant image samples: (1) Random offset transformation: The plant image sample is translated in different directions to increase the data volume and enhance the generalization of the data. In this embodiment, the image offset does not exceed 10%. (2) Grayscale transformation: This includes grayscale change and color saturation processing of the plant image sample to eliminate the interference of noise information that is not the research object and increase the model's recognition of the target image information features. (3) Random folding: In the process of random folding, the plant image sample is folded in three different ways, including horizontal, vertical, and oblique, to meet the recognition accuracy of the data within different transformation requirements and improve the generalization ability of data prediction. (4) Random scaling: In the case of random scaling, the random scaling ratio of the plant image sample is guaranteed to be no more than 10%. (5) Random stretching: The random stretching used in this embodiment refers to randomly stretching the plant image sample in the vertical or horizontal direction. The stretching ratio generally does not exceed 10%. (6) Random rotation: Random rotation refers to rotating the plant image sample by a certain proportion to increase the sample volume of the plant image sample and enhance the model generalization ability for complex site sample data.

[0054] In one embodiment, before inputting the labeled plant image samples into the backbone network to extract image features, the method further includes: adjusting the size of the labeled plant image samples.

[0055] Specifically, in order to unify the model input data, thereby facilitating the operation of the model, the influence of hardware factors is taken into account. The size of the marked plant image samples can be uniformly processed. The size of the marked plant image samples can be adjusted to the target size (for example, 224x224). When the marked plant image samples are resized, an image scaling method can be adopted. If the image size of the area less than 224x224 is filled with a value of 0, the resizing method in this embodiment retains the detail feature information in the original data image to a greater extent than using other methods such as cropping and filling. By using a scaling function in the later convolutional neural network recognition process, the loss of important information features in the subsequent vegetation information application process is avoided. After the size adjustment of the marked plant image samples is completed, the image sample format can be uniformly converted into the required format.

[0056] S2: The labeled plant image samples are input into the backbone network to extract image features, and the image features are optimized through the CBAM module to obtain the optimized feature map.

[0057] Please refer to Figure 3 The improved CBAM-MaskRCNN model mainly includes several parts, including the backbone network, CBAM module, RPN network, ROI Align network and fully connected layer. The backbone network is the Backbone part in the figure. The Backbone part is the main framework of the convolutional neural network. The CBAM attention mechanism is added in the middle to improve the segmentation accuracy of MaskRCNN. The attention mechanism CBAM part is composed of CA and SA. The CA part is the attention learning added to the pixel scale on the channel, and the SA is the attention learning added on the regional scale. RPN is the screening process of the feature region part, which is used for feature recognition. The ROI Align part resamples the feature information to a uniform size part, and finally obtains the feature map.

[0058] In one embodiment, the backbone network includes a ResNet network and a pyramid network; inputting the labeled plant image samples into the backbone network to extract image features includes: inputting the labeled plant image samples into the ResNet network for feature extraction to obtain initial features; and fusing the initial features through the pyramid network to obtain image features.

[0059] In one embodiment, the ResNet network includes a ResNet50 network or a ResNet101 network.

[0060] The backbone component can be a ResNet deep convolutional neural network. ResNet50 or ResNet101 are commonly used for feature extraction. In this embodiment, a 224x224 labeled plant image sample is input into ResNet, resulting in five feature maps of different sizes: 112x112, 56x56, 28x28, 14x14, and 7x7. Feature maps of different sizes contain different feature information, and different levels of feature information are identified through the convolutional neural network. This includes low-level details such as tree color, outline, and texture, as well as high-level features such as image category attributes. The improved backbone component also incorporates a feature pyramid network structure to fuse information features at different levels and prevent feature loss. The feature pyramid network structure simultaneously utilizes the spatial information of low-level feature maps and the semantic information of high-level feature maps. High-level features with lower resolution are first reduced in dimensionality through convolution, then upsampled to the same size as the previous feature map. Each element is then summed up element by element to obtain the fused features.

[0061] Furthermore, the improved CBAM-MaskRCNN architecture incorporates an attention mechanism into the Backbone layer, enhancing the model's training capabilities and improving segmentation accuracy. The CBAM layer consists of two components: CA (Channel Attention Mechanism) and SA (Spatial Attention Mechanism). CA increases learning capabilities at the pixel scale through channels, while SA increases learning capabilities at the regional spatial scale. The combined CBAM attention mechanism, built into the Backbone layer, enhances feature extraction accuracy.

[0062] S3: The optimized feature map is sequentially input into the RPN network and the ROI Align network for feature screening and resampling to obtain the final feature map.

[0063] The RPN (region recommendation) network, primarily responsible for filtering network features, uses a series of box points and then applies specific rules to filter feature information. The size of each box is determined by proportion and ratio. Each pixel can generate three boxes of different sizes. The area of the three pixel boxes remains unchanged. By changing the aspect ratio through scaling or ratio, three boxes of different sizes are generated. These three boxes allow for the selection of features of different sizes. Features at all scales can be recognized, enhancing boundary recognition and reducing the likelihood of missing subtle features.

[0064] The ROI Align part is an improvement over the CBAM-MaskRCNN model and other models. In this embodiment, for the ROI Align part, it is necessary to resample the categories of the different boxes generated by the RPN part to the same feature image size, and use the pooling layer to upsample the different feature categories into a one-dimensional vector to complete the subsequent classification and regression tasks. ROI Align performs bilinear interpolation on the sampling point pixels. The interpolated image is pooled for the sampling points within each cell to obtain the final ROI Align result.

[0065] S4: Input the final feature map into the fully connected layer to obtain the actual segmentation result.

[0066] Specifically, the category and positioning images output by the loss function pass through the fully connected layer, and finally output the single tree features of instance segmentation.

[0067] S5: Adjust the parameters of the improved CBAM-MaskRCNN model until the deviation between the actual segmentation result and the target segmentation result is within the allowable range. The training is completed and the plant segmentation model is obtained.

[0068] Specifically, the position of each plant in the marked plant image sample (i.e., the target segmentation result) can be determined based on the plant information, and then the actual segmentation result (i.e., the result obtained by segmenting the plant image sample using the improved CBAM-MaskRCNN model) is checked to see if its deviation is within the allowable range. If not, the weight of the improved CBAM-MaskRCNN model is adjusted and the actual segmentation result is re-output until the deviation between the actual segmentation result and the target segmentation result is within the allowable range. The training is completed and the plant segmentation model is obtained. The allowable range can be a preset value, indicating the maximum acceptable deviation between the actual segmentation result and the target segmentation result.

[0069] In addition, when training the improved CBAM-MaskRCNN model, its loss function is also improved. The description of the loss function is as follows:

[0070] In one embodiment, the plant segmentation method further includes: when training the improved CBAM-MaskRCNN model, the loss functions used include a regression loss function, a positioning loss function, and a mask loss function.

[0071] Specifically, if Figure 3As shown, the improved CBAM-MaskRCNN uses five loss functions. Since the attention-based CBAM-MaskRCNN model is multi-tasking, its loss function includes regression, localization, and segmentation (also known as masking). However, since the RPN architecture also performs regression and localization, a total of five loss functions are used. The improved CBAM-MaskRCNN loss function uses the MaskRCNN branch, which has outputs of different dimensions for each classified object. Lclass is the loss function used for classification in the corresponding CBAM-MaskRCNN model, which can be calculated by averaging the binary cross-entropy loss function. Lbox is the loss function used for localization in the corresponding CBAM-MaskRCNN model. Lmask is the loss function for the submerged mask component added to the CBAM-MaskRCNN model. This definition allows for the generation of masks for each class and eliminates inter-class competition. Ltotal is the overall loss function, which improves feature extraction accuracy and boundary range recognition to enhance prediction accuracy. Where Ltotal = Lclass + Lbox + Lmas, and Ltotal is the overall loss function of the improved CBAM-MaskRCNN model.

[0072] The single plant segmentation method in the embodiment of the present application uses labeled plant image samples to train the improved CBAM-MaskRCNN model to obtain a pre-trained plant segmentation model. Based on the plant segmentation model, single plants in the plant image to be identified can be identified, thereby realizing the segmentation of single plants, which is very convenient to use; and the improved CBAM-MaskRCNN model is used to add the CBAM attention module and the RPN network to the traditional convolutional model, which can realize the segmentation of small plants with high segmentation efficiency and accuracy.

[0073] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0074] The above embodiments disclosed in this application describe in detail a method for dividing a single plant. The above method disclosed in this application can be implemented using various forms of equipment. Therefore, this application also discloses a single plant dividing device corresponding to the above method. Specific embodiments are given below for detailed description.

[0075] See also Figure 4 , is a single plant segmentation device disclosed in an embodiment of the present application, mainly comprising:

[0076] The plant image acquisition module 410 is used to acquire the plant image to be identified; wherein the plant image to be identified includes one or more plants.

[0077] The segmentation result output module 420 is used to input the plant image to be identified into a pre-trained plant segmentation model to output the segmentation result of a single plant; wherein the plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples, and the improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROI Align network and a fully connected layer connected in sequence.

[0078] In one embodiment, the apparatus comprises:

[0079] The marking module is used to obtain plant image samples and mark each plant in the plant image samples using plant information.

[0080] The image feature extraction module is used to input the labeled plant image samples into the backbone network to extract image features.

[0081] The feature optimization module is used to optimize the image features through the CBAM module to obtain an optimized feature map.

[0082] The final feature map acquisition module is used to input the optimized feature map into the RPN network and ROI Align network in sequence for feature screening and resampling to obtain the final feature map;

[0083] The actual segmentation result acquisition module is used to input the final feature map into the fully connected layer to obtain the actual segmentation result;

[0084] The plant segmentation module acquisition module is used to adjust the parameters of the improved CBAM-MaskRCNN model until the deviation between the actual segmentation result and the target segmentation result is within the allowable range. The training is completed and the plant segmentation model is obtained.

[0085] In one embodiment, the backbone network includes a ResNet network and a pyramid network; the image feature extraction module is used to input the labeled plant image samples into the ResNet network for feature extraction to obtain initial features; the initial features are fused through the pyramid network to obtain image features.

[0086] In one embodiment, the ResNet network includes a ResNet50 network or a ResNet101 network.

[0087] In one embodiment, the apparatus further comprises:

[0088] The image enhancement module is used to randomly select one or more enhancement processing methods to enhance the marked plant image samples; when multiple enhancement processing methods are selected, the running order of each enhancement processing method is random, and the enhancement processing methods include grayscale transformation processing, random offset transformation processing, random folding processing, random scaling processing, random stretching processing and random rotation processing on the plant image samples.

[0089] In one embodiment, the device further includes: a size adjustment module, configured to adjust the size of the marked plant image sample.

[0090] In one embodiment, the device further includes: when training the improved CBAM-MaskRCNN model, the loss functions used include a regression loss function, a positioning loss function, and a mask loss function.

[0091] The specific definition of the device for segmenting a single plant can be found in the definition of the method above and will not be repeated here. Each module in the above-mentioned device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the terminal device in hardware form, or can be stored in the memory of the terminal device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0092] Please refer to Figure 5 , Figure 5 It shows a block diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 50 can be a computer device. The terminal device 50 in the present application can include one or more of the following components: a processor 52, a memory 54, and one or more application programs, wherein the one or more application programs can be stored in the memory 54 and configured to be executed by the one or more processors 52. The one or more application programs are configured to execute the method described in the above-mentioned method embodiment applied to the terminal device, and can also be configured to execute the method described in the above-mentioned method embodiment applied to the segmentation of a single plant.

[0093] The processor 52 may include one or more processing cores. The processor 52 utilizes various interfaces and circuits to connect various components within the terminal device 50. It executes instructions, programs, code sets, or instruction sets stored in the memory 54, and accesses data stored in the memory 54 to perform various functions and process data within the terminal device 50. Optionally, the processor 52 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 52 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 52 and may be implemented separately via a communication chip.

[0094] The memory 54 may include a random access memory (RAM) or a read-only memory (ROM). The memory 54 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 54 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described below, and the like. The data storage area may also store data created by the terminal device 50 during use.

[0095] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the terminal device to which the solution of the present application is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0096] In summary, the terminal device provided in the embodiment of the present application is used to implement the corresponding single plant segmentation method in the aforementioned method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0097] See also Figure 6, which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 60 stores program code, which can be invoked by a processor to execute the method described in the above-mentioned embodiment of the method for segmenting a single plant, or can be invoked by a processor to execute the method described in the above-mentioned embodiment of the method for segmenting a single plant.

[0098] The computer-readable storage medium 60 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 60 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 60 has storage space for program code 62 for executing any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 62 can be compressed, for example, in a suitable form.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0100] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for dividing a single plant, characterized in that: The method comprises: Acquire a plant image to be identified; wherein the plant image to be identified includes one or more plants; Inputting the plant image to be identified into a pre-trained plant segmentation model to output a single plant segmentation result; wherein the plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples, and the improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROIAlign network, and a fully connected layer connected in sequence; The training method of the plant segmentation model includes: Acquire a plant image sample, and mark each plant in the plant image sample using plant information; Inputting the labeled plant image samples into the backbone network to extract image features, and optimizing the image features through the CBAM module to obtain an optimized feature map; Inputting the optimized feature map into the RPN network and the ROI Align network in sequence for feature screening and resampling to obtain the final feature map; Input the final feature map into the fully connected layer to obtain the actual segmentation result; Adjusting the parameters of the improved CBAM-MaskRCNN model until the deviation between the actual segmentation result and the target segmentation result is within an allowable range, completing the training and obtaining the plant segmentation model; The backbone network includes a ResNet network and a pyramid network; and inputting the labeled plant image samples into the backbone network to extract image features includes: Inputting the labeled plant image samples into the ResNet network for feature extraction to obtain initial features; The initial features are subjected to feature fusion through the pyramid network to obtain the image features.

2. The method according to claim 1, characterized in that The ResNet network includes a ResNet50 network or a ResNet101 network.

3. The method according to claim 1 or 2, characterized in that Before marking each plant in the plant image sample using plant information, the method further includes: Randomly selecting one or more enhancement processing methods to perform enhancement processing on the plant image sample; When multiple enhancement processing methods are selected, the running order of each enhancement processing method is random, and the enhancement processing method includes grayscale transformation processing, random offset transformation processing, random folding processing, random scaling processing, random stretching processing and random rotation processing on the plant image sample.

4. The method according to claim 3, characterized in that Before inputting the labeled plant image samples into the backbone network to extract image features, the method further includes: The size of the marked plant image sample is adjusted.

5. The method according to claim 1 or 2, characterized in that The method further comprises: When training the improved CBAM-MaskRCNN model, the loss functions used include regression loss function, positioning loss function and mask loss function.

6. A device for dividing a single plant, characterized in that: The device comprises: A module for acquiring an image of a plant to be identified, configured to acquire an image of a plant to be identified; wherein the image of the plant to be identified includes one or more plants; A segmentation result module is used to input the plant image to be identified into a pre-trained plant segmentation model to output a single plant segmentation result; wherein the plant segmentation model is obtained by training an improved CBAM-MaskRCNN model using labeled plant image samples, and the improved CBAM-MaskRCNN model includes a backbone network, a CBAM module, an RPN network, a ROIAlign network and a fully connected layer connected in sequence; The device also includes: a marking module, configured to obtain a plant image sample and mark each plant in the plant image sample using plant information; An image feature extraction module; used to input the labeled plant image samples into the backbone network to extract image features, and optimize the image features through the CBAM module to obtain an optimized feature map; A feature optimization module is used to input the optimized feature map into the RPN network and the ROIAlign network in sequence for feature screening and resampling to obtain a final feature map; A final feature map acquisition module is used to input the final feature map into the fully connected layer to obtain an actual segmentation result; The final feature map acquisition module is used to adjust the parameters of the improved CBAM-MaskRCNN model until the deviation between the actual segmentation result and the target segmentation result is within the allowable range, and the training is completed to obtain the plant segmentation model; Among them, the backbone network includes a ResNet network and a pyramid network; the image feature extraction module is used to input the marked plant image samples into the ResNet network for feature extraction to obtain initial features; and the initial features are subjected to feature fusion through the pyramid network to obtain the image features.

7. A terminal device, characterized in that: include: Memory; one or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images