Plant three-dimensional point cloud processing method and device, electronic equipment and storage medium

Abstract

The present application provides a kind of plant three-dimensional point cloud processing method, device, electronic equipment and storage medium, wherein the method comprises: obtaining plant spectrum three-dimensional point cloud diagram;Plant spectrum three-dimensional point cloud diagram is input to classification network, and the plant disease point and plant non-disease point output by classification network are obtained;Classification network is used to extract plant disease point and plant non-disease point from plant spectrum three-dimensional point cloud diagram;Plant disease point and plant non-disease point are input to segmentation network, and the disease area segmentation graph output by segmentation network is obtained;Disease area segmentation graph is used to represent disease area and non-disease area in plant spectrum three-dimensional point cloud diagram, and segmentation network is used to segment disease area and non-disease area from plant spectrum three-dimensional point cloud diagram based on plant disease point and plant non-disease point, to obtain disease area segmentation graph.Compared with the related art through artificial visual assessment of plant disease, the present application embodiment can effectively improve the accuracy of evaluating plant disease.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and in particular to a plant three-dimensional point cloud processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] In agricultural production, plant diseases are one of the most important factors restricting crop yield, and the yield loss caused by diseases accounts for about 40% of the total yield worldwide each year. Therefore, in agricultural production, rapid detection and prediction of plant diseases are of great significance to reduce yield loss and achieve rapid growth of crop yield.

[0003] Currently, the disease severity of a single plant is usually determined by artificial visual inspection, combined with field investigation methods, and disease index is calculated by methods such as five-point sampling. When artificially evaluating the disease severity of a single plant, due to human subjective reasons and other fluctuations, the disease index obtained may have a large error, and the accuracy of evaluating plant diseases is low. SUMMARY

[0004] The present application provides a plant three-dimensional point cloud processing method and device, electronic equipment and storage medium to solve the problem of low accuracy in evaluating plant diseases in the prior art.

[0005] The present application provides a plant three-dimensional point cloud processing method, comprising:

[0006] obtaining a plant spectrum three-dimensional point cloud image;

[0007] inputting the plant spectrum three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network; the classification network is used to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image;

[0008] inputting the plant disease points and plant non-disease points into a segmentation network to obtain a disease area segmentation image output by the segmentation network; the disease area segmentation image is used to represent disease areas and non-disease areas in the plant spectrum three-dimensional point cloud image, and the segmentation network is used to segment disease areas and non-disease areas from the plant spectrum three-dimensional point cloud image based on the plant disease points and plant non-disease points to obtain the disease area segmentation image.

[0009] According to the plant three-dimensional point cloud processing method provided by the present application, before the plant spectrum three-dimensional point cloud image is input into the classification network to obtain plant disease points and plant non-disease points output by the classification network, the method further comprises:

[0010] inputting the plant spectrum three-dimensional point cloud image into a pre-set adversarial network to obtain a disease area point cloud image output by the adversarial network;

[0011] The plant spectrum three-dimensional point cloud graph is input into the classification network to obtain plant disease points and plant non-disease points output by the classification network.

[0012] The plant spectrum three-dimensional point cloud graph and the disease area point cloud graph are input into the classification network to obtain plant disease points and plant non-disease points output by the classification network.

[0013] According to the plant three-dimensional point cloud processing method provided by the application, the classification network comprises a preprocessing layer and a full connection layer.

[0014] The plant spectrum three-dimensional point cloud graph is input into the classification network to obtain plant disease points and plant non-disease points output by the classification network.

[0015] The plant spectrum three-dimensional point cloud graph is input into the preprocessing layer to obtain global feature values output by the preprocessing layer; the preprocessing layer is used to calculate global feature values corresponding to the plant spectrum three-dimensional point cloud graph.

[0016] The global feature values are input into the full connection layer to obtain plant disease points and plant non-disease points output by the full connection layer; the full connection layer is used to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud graph based on the global feature values.

[0017] According to the plant three-dimensional point cloud processing method provided by the application, the preprocessing layer comprises N abstract SA layers and N+1 convolution layers.

[0018] The plant spectrum three-dimensional point cloud graph is input into the preprocessing layer to obtain global feature values output by the preprocessing layer, comprising:

[0019] The plant spectrum three-dimensional point cloud graph is input into the N SA layers in the preprocessing layer, so that the plant spectrum three-dimensional point cloud graph sequentially passes through the N SA layers to obtain N extraction results respectively output by the N SA layers.

[0020] The plant spectrum three-dimensional point cloud graph and the N extraction results are input into the N+1 convolution layers to obtain N+1 convolution results output by the N+1 convolution layers.

[0021] Based on the N+1 convolution results and the extraction result output by the Nth SA layer in the N SA layers, the global feature values are obtained as the global feature values output by the preprocessing layer.

[0022] The processing method for a plant three-dimensional point cloud provided by the application further comprises N screening layers, and an output of a previous screening layer in the N screening layers is used as a first input of a subsequent screening layer in the N screening layers.

[0023] The global feature value is obtained based on the N+1 convolution results and the extraction result of the Nth SA layer in the N SA layers, and the global feature value is output by the preprocessing layer.

[0024] The convolution result output by the N+1th convolution layer is used as the first input of the first screening layer in the N screening layers, so that the convolution result output by the N+1th convolution layer sequentially passes through the N screening layers, and the convolution results output by the previous N convolution layers are used as the second inputs of the N screening layers, so that N screening results output by the N screening layers are obtained; the input of the N+1th convolution layer is the extraction result output by the Nth SA layer in the N SA layers.

[0025] The global feature value is obtained based on the N screening results and the extraction result of the Nth SA layer in the N SA layers, and the global feature value is output by the preprocessing layer.

[0026] In the processing method for a plant three-dimensional point cloud provided by the application, N is 4.

[0027] The application further provides a processing device for a plant three-dimensional point cloud, which comprises:

[0028] An acquisition module is configured to acquire a plant spectral three-dimensional point cloud image.

[0029] A classification module is configured to input the plant spectral three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network; the classification network is configured to extract the plant disease points and the plant non-disease points from the plant spectral three-dimensional point cloud image.

[0030] A segmentation module is configured to input the plant disease points and the plant non-disease points into a segmentation network to obtain a disease region segmentation image output by the segmentation network; the disease region segmentation image is configured to represent a disease region and a non-disease region in the plant spectral three-dimensional point cloud image, and the segmentation network is configured to segment the disease region and the non-disease region from the plant spectral three-dimensional point cloud image based on the plant disease points and the plant non-disease points to obtain the disease region segmentation image.

[0031] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the processing method for a plant three-dimensional point cloud according to any one of the above embodiments when executing the program.

[0032] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the plant three-dimensional point cloud processing method according to any one of the above.

[0033] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the plant three-dimensional point cloud processing method according to any one of the above.

[0034] The plant three-dimensional point cloud processing method, device, electronic equipment and storage medium provided by the application first acquire a plant spectrum three-dimensional point cloud image, extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image through a classification network, and then segment the plant spectrum three-dimensional point cloud image based on the plant disease points and the plant non-disease points through a segmentation network to obtain a disease area segmentation image representing a disease area and a non-disease area. Through the classification network and the segmentation network, the plant spectrum three-dimensional point cloud image can be effectively segmented into the disease area and the non-disease area, so that the plant disease can be evaluated according to the above areas. Compared with the manual visual evaluation of the plant disease in the related art, the embodiment of the application can effectively improve the accuracy of evaluating the plant disease. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0036] Figure 1 is one of the flowcharts of the plant three-dimensional point cloud processing method provided by the embodiment of the application;

[0037] Figure 2 is a schematic diagram of a plant spectrum three-dimensional point cloud image provided by the embodiment of the application;

[0038] Figure 3 is the second flowchart of the plant three-dimensional point cloud processing method provided by the embodiment of the application;

[0039] Figure 4 is the third flowchart of the plant three-dimensional point cloud processing method provided by the embodiment of the application;

[0040] Figure 5 is the fourth flowchart of the plant three-dimensional point cloud processing method provided by the embodiment of the application;

[0041] Figure 6is a structural schematic diagram of a screening layer in a plant three-dimensional point cloud processing method provided by an embodiment of the present application.

[0042] Figure 7 is a schematic diagram of a disease area segmentation result of the plant three-dimensional point cloud processing method provided by an embodiment of the present application.

[0043] Figure 8 is a structural schematic diagram of a plant three-dimensional point cloud processing device provided by an embodiment of the present application.

[0044] Figure 9 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described below in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0046] The plant three-dimensional point cloud processing method, device, electronic device, and storage medium of the present application will be described below in conjunction with the accompanying drawings.

[0047] Figure 1 is one of flowcharts of a plant three-dimensional point cloud processing method provided by an embodiment of the present application, as shown in the figure, the method comprises steps 101 to 103; wherein: Figure 1

[0048] Step 101, acquiring a plant spectrum three-dimensional point cloud image;

[0049] Step 102, inputting the plant spectrum three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network;

[0050] The classification network is used to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image.

[0051] Step 103, inputting the plant disease points and plant non-disease points into a segmentation network to obtain a disease area segmentation image output by the segmentation network;

[0052] The disease area segmentation image is used to represent disease areas and non-disease areas in the plant spectrum three-dimensional point cloud image, and the segmentation network is used to segment disease areas and non-disease areas from the plant spectrum three-dimensional point cloud image based on the plant disease points and plant non-disease points to obtain the disease area segmentation image.

[0053] ​Specifically, in the related art, the disease occurrence severity of a single plant is usually determined by artificial visual inspection, and a disease index is calculated by combining field investigation methods, such as five-point sampling. When the disease occurrence severity of a single plant is evaluated by artificial visual inspection, there is fluctuation due to subjective reasons of a person, and the disease index obtained may have a large error, and the accuracy of evaluating plant diseases is low.

[0054] The plant spectrum three-dimensional point cloud graph is input into a classification network, plant disease points and plant non-disease points are extracted from the plant spectrum three-dimensional point cloud graph by the classification network, and the plant disease points and the plant non-disease points output by the classification network are obtained. The plant disease points and the plant non-disease points are input into a segmentation network, and the disease area and the non-disease area are segmented from the plant spectrum three-dimensional point cloud graph based on the plant disease points and the plant non-disease points by the segmentation network, and a disease area segmentation graph output by the segmentation network for representing the disease area and the non-disease area in the plant spectrum three-dimensional point cloud graph is obtained. Compared with evaluating plant diseases by artificial visual inspection, there is fluctuation due to subjective reasons of a person, and the disease index obtained may have a large error. In the embodiment of the present application, the plant spectrum three-dimensional point cloud graph is sequentially input into the classification network and the segmentation network, and the disease area segmentation graph representing the disease area and the non-disease area in the plant spectrum three-dimensional point cloud graph can be obtained. The disease area segmentation graph can more truly reflect the plant disease index, and the accuracy of evaluating plant diseases can be improved. Moreover, the plant disease index can be more truly reflected, and the research on resistant variety screening is also facilitated.

[0055] Optionally, the plant three-dimensional point cloud can be reconstructed based on the surrounding spectral image sequence, and a plant spectrum three-dimensional point cloud graph is obtained, Figure 2 is a schematic diagram of the plant spectrum three-dimensional point cloud graph provided by the embodiment of the present application, as Figure 2 As shown in the figure, the plant spectrum three-dimensional point cloud graph realizes the combination of the plant three-dimensional point cloud and the spectrum, and there is a one-to-one mapping relationship between the plant three-dimensional point cloud and the spectrum. Each plant three-dimensional point cloud can correspond to n-dimensional spectral information.

[0056] It should be noted that the remote sensing means of multispectral or hyperspectral can overcome the limitation of the visible light band, and compared with artificial visual recognition, the physiological condition of the plant can be detected in advance, and the monitoring range can be expanded. However, the remote sensing means based on multispectral and hyperspectral currently compresses all information to a two-dimensional plane, and cannot truly reflect the changes in the three-dimensional space of the plant. The existing method of evaluating plant diseases by using machine vision mainly trains and verifies the two-dimensional photos of plants with diseases. Although this method can objectively and accurately segment the diseases and classify the disease severity to a certain extent, the accuracy of disease recognition and prediction, especially early prediction, is still low.

[0057] The embodiment of the present application reconstructs a plant three-dimensional point cloud based on a surrounding spectral image sequence, obtains a plant spectral three-dimensional point cloud map, and obtains a disease area segmentation map representing a disease area and a non-disease area in the plant spectral three-dimensional point cloud map based on the plant spectral three-dimensional point cloud map. The disease area segmentation map obtained in this way can truly reflect the changes in the three-dimensional space of the plant, and improves the identification and prediction of plant diseases, especially the accuracy of early prediction.

[0058] It should be further noted that, since a plant is an irregular three-dimensional object with a complex structure, it is difficult to provide complete information of the plant in the three-dimensional space based on spectral information only, and it is also difficult to map the real situation of the plant under disease stress. In addition, early plant diseases have small and sparse disease spots and are often located in lower leaves. The embodiment of the present application reconstructs a plant three-dimensional point cloud based on a surrounding spectral image sequence, obtains a plant spectral three-dimensional point cloud map, and combines three-dimensional information and spectral information to fully exploit the complementary information of multi-source data, so as to monitor the spatio-temporal distribution rule of crop diseases and more accurately monitor small and sparse disease spots, thereby achieving precise prevention and control of early plant diseases.

[0059] Optionally, the classification network and the segmentation network in the PointNet++ deep learning network can be used as the classification network and the segmentation network in the embodiment of the present application.

[0060] In the embodiment of the present application, a plant spectral three-dimensional point cloud map is first obtained, plant disease points and plant non-disease points are extracted from the plant spectral three-dimensional point cloud map through a classification network, and a disease area segmentation map representing a disease area and a non-disease area is obtained by segmenting the plant spectral three-dimensional point cloud map based on the plant disease points and the plant non-disease points through a segmentation network. Through the classification network and the segmentation network, the plant spectral three-dimensional point cloud map can be effectively segmented into a disease area and a non-disease area, so as to evaluate plant diseases according to the above areas. Compared with the manual visual evaluation of plant diseases in the related art, the embodiment of the present application can effectively improve the accuracy of evaluating plant diseases.

[0061] Optionally, Figure 3 is a flowchart of a second plant three-dimensional point cloud processing method according to an embodiment of the present application, as shown in Figure 3 The method comprises steps 301 to 304; wherein:

[0062] Step 301, obtaining a plant spectral three-dimensional point cloud map;

[0063] Step 302, inputting the plant spectral three-dimensional point cloud map into a pre-set adversarial network to obtain a disease area point cloud map output by the adversarial network;

[0064] Step 303, inputting the plant spectrum three-dimensional point cloud graph and the disease area point cloud graph into the classification network to obtain plant disease points and plant non-disease points output by the classification network;

[0065] The classification network is used to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud graph.

[0066] Step 304, inputting the plant disease points and plant non-disease points into a segmentation network to obtain a disease area segmentation graph output by the segmentation network.

[0067] The disease area segmentation graph is used to represent disease areas and non-disease areas in the plant spectrum three-dimensional point cloud graph, and the segmentation network is used to segment disease areas and non-disease areas from the plant spectrum three-dimensional point cloud graph based on the plant disease points and plant non-disease points to obtain the disease area segmentation graph.

[0068] It should be noted that the specific implementation of steps 301 and 304 can refer to the specific implementation of steps 101 and 103 described above, and will not be repeated here.

[0069] Specifically, the plant spectrum three-dimensional point cloud graph is obtained, and the plant spectrum three-dimensional point cloud graph is first input into a pre-set adversarial network for preliminary processing of the plant spectrum three-dimensional point cloud graph by the adversarial network, specifically, the disease area in the plant spectrum three-dimensional point cloud graph is preliminarily distinguished to obtain a disease area point cloud graph output by the adversarial network, and then the plant spectrum three-dimensional point cloud graph and the obtained disease area point cloud Figure 1 graph are input into the classification network to increase the breadth of the disease area features in the original data.

[0070] In the embodiment of the application, the plant spectrum three-dimensional point cloud graph is preliminarily processed by the adversarial network to obtain a disease area point cloud graph output by the adversarial network, so that the plant spectrum three-dimensional point cloud graph and the disease area point cloud graph are both input into the classification network, effectively increasing the amount of input data of the classification network, i.e. increasing the breadth of the disease area features in the original data, which is conducive to the classification network to better learn data and classification, and can improve the classification accuracy of the classification network, and thus can improve the segmentation effect of the disease area and the non-disease area of the plant.

[0071] Optionally, the classification network comprises a preprocessing layer and a fully connected layer. Figure 4 is a flowchart of a plant three-dimensional point cloud processing method provided by an embodiment of the application, as shown in the figure, the method comprises steps 401 to 404; wherein: Figure 4

[0072] Step 401, obtaining a plant spectrum three-dimensional point cloud graph;

[0073] ​Step 402, input the plant spectrum three-dimensional point cloud image into the preprocessing layer to obtain global feature values output by the preprocessing layer;

[0074] The preprocessing layer is configured to calculate global feature values corresponding to the plant spectrum three-dimensional point cloud image.

[0075] Step 403, input the global feature values into the fully connected layer to obtain plant disease points and plant non-disease points output by the fully connected layer.

[0076] The fully connected layer is configured to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image based on the global feature values.

[0077] Step 404, input the plant disease points and plant non-disease points into the segmentation network to obtain a disease area segmentation map output by the segmentation network.

[0078] The disease area segmentation map is configured to represent disease areas and non-disease areas in the plant spectrum three-dimensional point cloud image, and the segmentation network is configured to segment disease areas and non-disease areas from the plant spectrum three-dimensional point cloud image based on the plant disease points and plant non-disease points to obtain the disease area segmentation map.

[0079] It should be noted that the specific implementation of steps 401 and 404 can refer to the specific implementation of steps 101 and 103 described above, which will not be repeated here.

[0080] Specifically, the classification network can include a preprocessing layer and a fully connected layer. First, the plant spectrum three-dimensional point cloud image is input into the preprocessing layer to extract features from the plant spectrum three-dimensional point cloud image by the preprocessing layer, to obtain global feature values corresponding to the plant spectrum three-dimensional point cloud image output by the preprocessing layer. Then, the global feature values corresponding to the plant spectrum three-dimensional point cloud image are input into the fully connected layer to classify based on the global feature values by the fully connected layer, to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image, to obtain plant disease points and plant non-disease points output by the fully connected layer. Then, the segmentation network segments disease areas and non-disease areas in the plant spectrum three-dimensional point cloud image based on the obtained plant disease points and plant non-disease points, to obtain the disease area segmentation map.

[0081] Optionally, the preprocessing layer can include N set abstraction (SA) layers and N+1 convolution layers.

[0082] The inputting of the plant spectrum three-dimensional point cloud image into the preprocessing layer to obtain the global feature values output by the preprocessing layer includes:

[0083] input the plant spectrum three-dimensional point cloud graph into N SA layers in the preprocessing layer, so that the plant spectrum three-dimensional point cloud graph sequentially passes through the N SA layers, and N extraction results output by the N SA layers are obtained;

[0084] input the plant spectrum three-dimensional point cloud graph and the N extraction results into the N+1 convolution layers, and obtain N+1 convolution results output by the N+1 convolution layers;

[0085] based on the N+1 convolution results and the extraction result output by the Nth SA layer in the N SA layers, obtain a global feature value as a global feature value output by the preprocessing layer.

[0086] Optionally, the value of N can be 4.

[0087] The working principle of the preprocessing layer in the plant three-dimensional point cloud processing method provided in the embodiment of the application will be described below by taking the value of N as 4.

[0088] Figure 5 is a fourth flowchart of the plant three-dimensional point cloud processing method provided in the embodiment of the application, as shown in the figure, the preprocessing layer includes four SA layers (SA layer 1, SA layer 2, SA layer 3 and SA layer 4) and five convolution layers. Figure 5

[0089] inputting the plant spectrum three-dimensional point cloud graph into the preprocessing layer is actually inputting the plant spectrum three-dimensional point cloud graph into the SA layer 1 in the preprocessing layer, and the plant spectrum three-dimensional point cloud graph can be expressed as a mathematical formula n i ×(d+C i ), wherein n i represents the number of concentrated points of the i-th layer point, d represents the coordinate dimension of the point, and C i represents other feature dimensions of the i-th layer point. The original input data is recorded as the 0th layer and is marked as n×(d+C). Input n×(d+C) into the SA layer 1 to obtain the extraction result n1×(d+C1) output by the SA layer 1, input the obtained n1×(d+C1) as the input of the SA layer 2 to obtain the extraction result n2×(d+C2) output by the SA layer 2, input the obtained n2×(d+C2) as the input of the SA layer 3 to obtain the extraction result n3×(d+C3) output by the SA layer 3, and finally input the obtained n3×(d+C3) as the input of the SA layer 4 to obtain the extraction result n4×(d+C4) output by the SA layer 4.

[0090] ​n×(d+C), n1×(d+C1), n2×(d+C2), n3×(d+C3) and n4×(d+C4) are taken as the inputs of the five convolutional layers respectively, and the convolutional results (n×C4, n1×C4, n2×C4, n3×C4 and n4×C4) output by the five convolutional layers and the extraction result n4×(d+C4) output by the SA layer 4 are integrated by Max Pooling to obtain the global feature value as the global feature value output by the preprocessing layer.

[0091] In the embodiment of the application, a specific implementation of the classification network is provided. Compared with the classification network in the existing PointNet++ deep learning network, the embodiment of the application increases the number of SA layers to improve the feature extraction effect of the preprocessing layer, and further improves the complexity of the classification network and the classification accuracy of the plant disease points and non-plant disease points, and improves the accuracy of evaluating plant diseases. Further, the value of N can be 4, and when the number of SA layers is 4, the complexity of the preprocessing layer and the learning depth of the preprocessing layer can be considered, that is, the complexity of the classification network and the classification accuracy of the plant disease points and non-plant disease points can be considered.

[0092] Optionally, the preprocessing layer further comprises N screening layers, and the output of a previous screening layer in the N screening layers is taken as the first input of a subsequent screening layer in the N screening layers.

[0093] The global feature value is obtained based on the N+1 convolutional results and the extraction result output by the Nth SA layer in the N SA layers, and the global feature value is taken as the global feature value output by the preprocessing layer, comprising:

[0094] The convolutional result output by the N+1th convolutional layer is taken as the first input of the first screening layer in the N screening layers, so that the convolutional result output by the N+1th convolutional layer passes through the N screening layers in turn, and the convolutional results output by the first N convolutional layers are taken as the second inputs of the N screening layers, to obtain N screening results output by the N screening layers; the input of the N+1th convolutional layer is the extraction result output by the Nth SA layer in the N SA layers.

[0095] The global feature value is obtained based on the N screening results and the extraction result output by the Nth SA layer in the N SA layers, and the global feature value is taken as the global feature value output by the preprocessing layer.

[0096] Specifically, please refer to Figure 5 The preprocessing layer can further comprise N screening layers.

[0097] The working principle of the preprocessing layer in the plant three-dimensional point cloud processing method provided by the embodiment of the application will be described below taking the value of N as 4.

[0098] In the embodiment corresponding to the pre-processing layer including 4 SA layers and 5 convolution layers, the embodiment further includes 4 screening layers.

[0099] In the embodiment, the convolution results (n x C4, n1 x C4, n2 x C4, n3 x C4) output by the first four convolution layers are taken as the second inputs of the four screening layers, and the convolution result (n4 x C4) output by the fifth convolution layer is taken as the first input of the first screening layer (the rightmost screening layer in the figure), so that the convolution result output by the fifth convolution layer sequentially passes through the four screening layers, and after being screened by the first screening layer, the screening result output by the first screening layer is taken as the first input of the second screening layer for further screening, and the screening result output by the first screening layer is retained, and so on, and the four screening layers can obtain four screening results, and the four screening results and the extraction result n4 x (d + C4) output by the SA layer 4 are taken through Max Pooling to obtain the global feature value as the global feature value output by the pre-processing layer.

[0100] Optionally, the screening layer can be a residual network feature screening layer.

[0101] Figure 6 is a structural schematic diagram of a screening layer in the plant three-dimensional point cloud processing method provided by the embodiment, as shown in Figure 6 The screening layer includes upsampling, data superposition and a residual network, the input of the upsampling is taken as the first input of the screening layer, the output of the upsampling is taken as the first input of the data superposition, the second input of the data superposition is taken as the second input of the screening layer, the output of the data superposition is taken as the input of the residual network, and the output of the residual network is taken as the output of the screening layer.

[0102] In the embodiment, the feature information extracted through the SA layer is no longer directly input into the full connection layer for classification, but the intermediate features (extraction results) output by each layer are connected to the newly constructed screening layer after passing through the convolution layer, so as to screen the feature values corresponding to the extracted feature information, and then input the screened feature values into the full connection layer for classification after merging the screened feature values into global feature values, so as to comprehensively utilize the feature information of each layer, further strengthen the feature extraction capability of the classification network, and thus improve the complexity of the classification network and the classification accuracy of the plant disease points and non-plant disease points, and improve the accuracy of evaluating plant diseases.

[0103] The plant three-dimensional point cloud processing method provided by the embodiment will be described below.

[0104] The embodiment of the present application first reconstructs a plant three-dimensional point cloud based on a surrounding spectral image sequence to obtain a plant spectral three-dimensional point cloud image, and realizes one-to-one mapping of the spectral and point cloud of the three-dimensional point cloud and the spectrum. Each three-dimensional point cloud corresponds to n-dimensional spectral information of a wave band.

[0105] Then, based on a PointNet++ deep learning network framework, a three-dimensional spectral point cloud of a plant disease spectrum index and a structure parameter is fused to construct a new multi-layer deep learning segmentation network. The new multi-layer deep learning segmentation network includes a classification network and a segmentation network. The classification network includes a preprocessing layer and a full connection layer. The preprocessing layer includes four SA layers, five convolution layers and four screening layers. The specific working principle is referred to the above method embodiment, which will not be repeated here. The plant disease spectrum index includes, for example, a normalized difference vegetation index (NDVI).

[0106] For the input plant spectral three-dimensional point cloud image, an adversarial network is designed to preliminarily process the original plant spectral three-dimensional point cloud image to initially identify the disease area. Then, the original plant spectral three-dimensional point cloud image and the preliminary identification result (disease area point cloud image output by the adversarial network) processed by the adversarial network are input into the classification network to increase the breadth of the disease area features in the original data.

[0107] The embodiment of the present application improves the bottom-up feature extraction structure of the PointNet++ backbone network. The improved new network builds a top-down residual network feature screening layer beside its branch to comprehensively utilize the features of each layer and further strengthen the feature extraction capability of the classification network.

[0108] In addition, the embodiment of the present application increases two SA layers on the basis of the original classification network to improve the feature extraction effect of the plant disease area. The feature information extracted by the SA layer is no longer directly input into the full connection layer for classification, but the intermediate features output by each layer are connected to the newly constructed residual network feature screening layer after the convolution layer, and then the feature values output by each screening layer are combined together and input into the full connection layer to obtain global feature values.

[0109] Then, the global feature values are input into the segmentation network to obtain a disease area segmentation image representing the disease area and the non-disease area in the plant spectral three-dimensional point cloud image.

[0110] The embodiment of the present application screens a plurality of vegetation indices significantly related to plant diseases, and preliminarily segments the disease area and the non-disease area (healthy area) based on the plant three-dimensional point cloud processing method provided by the embodiment of the present application.

[0111] Figure 7is a schematic diagram of a disease area segmentation result of a plant three-dimensional point cloud processing method provided by an embodiment of the present application, as shown in Figure 7 The first column is a tobacco virus RGB three-dimensional image, and the second column is a vegetation index. The first row is a graph corresponding to a plant with severe disease, and the second row is a graph corresponding to a plant with slight disease.

[0112] The plant three-dimensional point cloud processing device provided by the present application is described below. The plant three-dimensional point cloud processing device described below can be referred to in correspondence with the plant three-dimensional point cloud processing method described above.

[0113] Figure 8 is a structural schematic diagram of a plant three-dimensional point cloud processing device provided by an embodiment of the present application, as shown in Figure 8 The plant three-dimensional point cloud processing device 800 includes:

[0114] The acquisition module 801 is configured to acquire a plant spectral three-dimensional point cloud graph.

[0115] The classification module 802 is configured to input the plant spectral three-dimensional point cloud graph into a classification network to obtain plant disease points and plant non-disease points output by the classification network.

[0116] The classification network is configured to extract the plant disease points and the plant non-disease points from the plant spectral three-dimensional point cloud graph.

[0117] The segmentation module 803 is configured to input the plant disease points and the plant non-disease points into a segmentation network to obtain a disease area segmentation graph output by the segmentation network.

[0118] The disease area segmentation graph is configured to represent disease areas and non-disease areas in the plant spectral three-dimensional point cloud graph. The segmentation network is configured to segment the disease areas and the non-disease areas from the plant spectral three-dimensional point cloud graph based on the plant disease points and the plant non-disease points to obtain the disease area segmentation graph.

[0119] In the embodiment of the present application, the acquisition module acquires a plant spectral three-dimensional point cloud graph, and the classification module extracts plant disease points and plant non-disease points from the plant spectral three-dimensional point cloud graph through a classification network. Then, the segmentation module segments the plant spectral three-dimensional point cloud graph based on the plant disease points and the plant non-disease points through a segmentation network to obtain a disease area segmentation graph representing disease areas and non-disease areas. Through the classification network and the segmentation network, the plant spectral three-dimensional point cloud graph can be effectively segmented into disease areas and non-disease areas to evaluate plant diseases according to the above areas. Compared with the manual visual evaluation of plant diseases in the related art, the embodiment of the present application can effectively improve the accuracy of evaluating plant diseases.

[0120] Optionally, the plant three-dimensional point cloud processing apparatus 800 further comprises:

[0121] a processing module, configured to input the plant spectrum three-dimensional point cloud image into a preset adversarial network to obtain a disease area point cloud image output by the adversarial network;

[0122] The classification module 802 is specifically configured to input the plant spectrum three-dimensional point cloud image and the disease area point cloud image into the classification network to obtain plant disease points and plant non-disease points output by the classification network.

[0123] Optionally, the classification network comprises a preprocessing layer and a fully connected layer.

[0124] The classification module 802 is further specifically configured to:

[0125] input the plant spectrum three-dimensional point cloud image into the preprocessing layer to obtain global feature values output by the preprocessing layer; the preprocessing layer is configured to calculate global feature values corresponding to the plant spectrum three-dimensional point cloud image;

[0126] input the global feature values into the fully connected layer to obtain plant disease points and plant non-disease points output by the fully connected layer; the fully connected layer is configured to extract plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image based on the global feature values.

[0127] Optionally, the preprocessing layer comprises N abstract SA layers and N+1 convolution layers.

[0128] The classification module 802 is further specifically configured to:

[0129] input the plant spectrum three-dimensional point cloud image into the N SA layers in the preprocessing layer, so that the plant spectrum three-dimensional point cloud image sequentially passes through the N SA layers to obtain N extraction results respectively output by the N SA layers;

[0130] input the plant spectrum three-dimensional point cloud image and the N extraction results into the N+1 convolution layers in correspondence to obtain N+1 convolution results output by the N+1 convolution layers;

[0131] obtain global feature values as the global feature values output by the preprocessing layer based on the N+1 convolution results and an extraction result output by an Nth SA layer in the N SA layers.

[0132] Optionally, the preprocessing layer further comprises N screening layers, and an output of a previous screening layer in the N screening layers is taken as a first input of a subsequent screening layer in the N screening layers.

[0133] The classification module 802 is further specifically configured to:

[0134] The convolution result output by the N+1th convolution layer is taken as a first input of a first screening layer in the N screening layers, so that the convolution result output by the N+1th convolution layer sequentially passes through the N screening layers, and the convolution results output by the previous N convolution layers are taken as second inputs of the N screening layers, to obtain N screening results output by the N screening layers; the input of the N+1th convolution layer is the extraction result output by the Nth SA layer in the N SA layers;

[0135] Based on the N screening results and the extraction result output by the Nth SA layer in the N SA layers, a global feature value is obtained as a global feature value output by the preprocessing layer.

[0136] Optionally, the value of N is 4.

[0137] Figure 9 is a structural schematic diagram of an electronic device provided by the application, as Figure 9 shown, the electronic device 900 can include: a processor 910, a communications interface 920, a memory 930 and a communications bus 940, wherein the processor 910, the communications interface 920, the memory 930 complete mutual communication through the communications bus 940. The processor 910 can invoke the logical instructions in the memory 930 to execute the processing method of the plant three-dimensional point cloud, the method comprising:

[0138] Obtaining a plant spectral three-dimensional point cloud image;

[0139] Inputting the plant spectral three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network; the classification network is used for extracting plant disease points and plant non-disease points from the plant spectral three-dimensional point cloud image;

[0140] Inputting the plant disease points and plant non-disease points into a segmentation network to obtain a disease area segmentation map output by the segmentation network; the disease area segmentation map is used for representing disease areas and non-disease areas in the plant spectral three-dimensional point cloud image, and the segmentation network is used for segmenting disease areas and non-disease areas from the plant spectral three-dimensional point cloud image based on the plant disease points and plant non-disease points, to obtain the disease area segmentation map.

[0141] Further, the logic instructions in the memory 930 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0142] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the plant three-dimensional point cloud processing method provided by the above-mentioned methods, and the method comprises:

[0143] obtaining a plant spectral three-dimensional point cloud image;

[0144] inputting the plant spectral three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network; the classification network is used for extracting plant disease points and plant non-disease points from the plant spectral three-dimensional point cloud image;

[0145] inputting the plant disease points and the plant non-disease points into a segmentation network to obtain a disease area segmentation image output by the segmentation network; the disease area segmentation image is used for representing disease areas and non-disease areas in the plant spectral three-dimensional point cloud image, and the segmentation network is used for segmenting disease areas and non-disease areas from the plant spectral three-dimensional point cloud image based on the plant disease points and the plant non-disease points to obtain the disease area segmentation image.

[0146] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the plant three-dimensional point cloud processing method provided by the above-mentioned methods, and the method comprises:

[0147] obtaining a plant spectral three-dimensional point cloud image;

[0148] input the plant disease points and the plant non-disease points into a segmentation network to obtain a disease area segmentation map output by the segmentation network; the disease area segmentation map is used to represent disease areas and non-disease areas in the plant spectral three-dimensional point cloud map, and the segmentation network is used to segment the disease areas and the non-disease areas from the plant spectral three-dimensional point cloud map based on the plant disease points and the plant non-disease points to obtain the disease area segmentation map.

[0149] input the plant disease points and the plant non-disease points into a segmentation network to obtain a disease area segmentation map output by the segmentation network; the disease area segmentation map is used to represent disease areas and non-disease areas in the plant spectral three-dimensional point cloud map, and the segmentation network is used to segment the disease areas and the non-disease areas from the plant spectral three-dimensional point cloud map based on the plant disease points and the plant non-disease points to obtain the disease area segmentation map.

[0150] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0151] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0152] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of processing a plant three-dimensional point cloud, characterized in that, The method comprises the following steps: obtaining a plant spectrum three-dimensional point cloud image; inputting the plant spectrum three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network; the classification network is used for extracting plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image; inputting the plant disease points and the plant non-disease points into a segmentation network to obtain a disease area segmentation image output by the segmentation network; the disease area segmentation image is used for representing disease areas and non-disease areas in the plant spectrum three-dimensional point cloud image, and the segmentation network is used for segmenting disease areas and non-disease areas from the plant spectrum three-dimensional point cloud image based on the plant disease points and the plant non-disease points to obtain the disease area segmentation image; the classification network comprises a preprocessing layer and a full connection layer; the step of inputting the plant spectrum three-dimensional point cloud image into the classification network to obtain plant disease points and plant non-disease points output by the classification network comprises the following steps: inputting the plant spectrum three-dimensional point cloud image into the preprocessing layer to obtain global feature values output by the preprocessing layer; the preprocessing layer is used for calculating global feature values corresponding to the plant spectrum three-dimensional point cloud image; inputting the global feature values into the full connection layer to obtain plant disease points and plant non-disease points output by the full connection layer; the full connection layer is used for extracting plant disease points and plant non-disease points from the plant spectrum three-dimensional point cloud image based on the global feature values; the preprocessing layer comprises N abstract SA layers and N+1 convolution layers; the step of inputting the plant spectrum three-dimensional point cloud image into the preprocessing layer to obtain global feature values output by the preprocessing layer comprises the following steps: inputting the plant spectrum three-dimensional point cloud image into N SA layers in the preprocessing layer, so that the plant spectrum three-dimensional point cloud image sequentially passes through the N SA layers to obtain N extraction results respectively output by the N SA layers; inputting the plant spectrum three-dimensional point cloud image and the N extraction results into N+1 convolution layers to obtain N+1 convolution results output by the N+1 convolution layers; based on the N+1 convolution results and an extraction result output by an Nth SA layer in the N SA layers, global feature values are obtained as the global feature values output by the preprocessing layer.

2. The method of claim 1, wherein, Before the step of inputting the plant spectrum three-dimensional point cloud image into the classification network to obtain plant disease points and plant non-disease points output by the classification network, the method further comprises the following steps: inputting the plant spectrum three-dimensional point cloud image into a pre-set adversarial network to obtain a disease area point cloud image output by the adversarial network; the step of inputting the plant spectrum three-dimensional point cloud image into the classification network to obtain plant disease points and plant non-disease points output by the classification network comprises the following steps: inputting the plant spectrum three-dimensional point cloud image and the disease area point cloud image into the classification network to obtain plant disease points and plant non-disease points output by the classification network.

3. The method of claim 1, wherein, the preprocessing layer further comprises N screening layers, and an output of a previous screening layer in the N screening layers is used as a first input of a subsequent screening layer in the N screening layers. The obtaining, based on the N+1 convolution results and the extraction result output by the Nth SA layer among the N SA layers, a global eigenvalue as the global eigenvalue output by the preprocessing layer, includes: The convolution result output by the N+1th convolution layer is used as the first input of the first screening layer among the N screening layers, so that the convolution result output by the N+1th convolution layer passes through the N screening layers in sequence, and the convolution results output by the first N convolution layers are used as the second input of the N screening layers, thereby obtaining N screening results output by the N screening layers; the input of the N+1th convolution layer is the extraction result output by the Nth SA layer among the N SA layers; Based on the N screening results and the extraction result output by the Nth SA layer among the N SA layers, a global feature value is obtained as the global feature value output by the preprocessing layer.

4. The method of processing a plant three-dimensional point cloud according to claim 3, wherein, The value of N is 4.

5. A processing device of a plant three-dimensional point cloud, characterized in that, include: Acquisition module, used to obtain plant spectrum three-dimensional point cloud map; A classification module, configured to input the plant spectral three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network; the classification network is configured to extract plant disease points and plant non-disease points from the plant spectral three-dimensional point cloud image; a segmentation module, configured to input the plant disease points and the plant non-disease points into a segmentation network to obtain a diseased region segmentation map output by the segmentation network; the diseased region segmentation map is used to characterize the diseased region and the non-disease region in the plant spectral three-dimensional point cloud map; the segmentation network is configured to segment the diseased region and the non-disease region from the plant spectral three-dimensional point cloud map based on the plant disease points and the plant non-disease points to obtain the diseased region segmentation map; The classification network includes a preprocessing layer and a fully connected layer; The step of inputting the plant spectral three-dimensional point cloud image into a classification network to obtain plant disease points and plant non-disease points output by the classification network comprises: Inputting the plant spectral three-dimensional point cloud image into the preprocessing layer to obtain the global eigenvalue output by the preprocessing layer; the preprocessing layer is used to calculate the global eigenvalue corresponding to the plant spectral three-dimensional point cloud image; Inputting the global eigenvalues ​​into the fully connected layer to obtain plant disease points and plant non-disease points output by the fully connected layer; the fully connected layer is used to extract plant disease points and plant non-disease points from the plant spectral three-dimensional point cloud image based on the global eigenvalues; The preprocessing layer includes N abstract SA layers and N+1 convolutional layers; The step of inputting the plant spectral three-dimensional point cloud image into the preprocessing layer to obtain the global eigenvalue output by the preprocessing layer includes: Inputting the plant spectral three-dimensional point cloud image into N SA layers in the preprocessing layer, so that the plant spectral three-dimensional point cloud image passes through the N SA layers in sequence, and obtaining N extraction results output by each of the N SA layers; Inputting the plant spectrum three-dimensional point cloud image and the N extraction results into the N+1 convolution layers accordingly, and obtaining N+1 convolution results output by the N+1 convolution layers; Based on the N+1 convolution results and the extraction result output by the Nth SA layer in the N SA layers, a global feature value is obtained as a global feature value output by the preprocessing layer.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the plant three-dimensional point cloud processing method of any one of claims 1-4 when executing the program.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the plant three-dimensional point cloud processing method of any one of claims 1-4 when executed by the processor.

8. A computer program product comprising a computer program, characterized in that, The computer program implements the plant three-dimensional point cloud processing method of any one of claims 1-4 when executed by the processor. The computer program implements the plant three-dimensional point cloud processing method of any one of claims 1-4 when executed by the processor.

Images