Ground segmentation learning model training method, ground plane determination method and related device

Through the ground segmentation learning model training method, point cloud data and standard ground surfaces are used to identify ground planes, which solves the problem that drones cannot accurately sense the location of surface objects in farmlands, and improves the obstacle avoidance and safety of drones .

CN112329946BActive Publication Date: 2024-12-03GUANGZHOU XAIRCRAFT TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202011246136.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-10
Publication Date
2024-12-03
Estimated Expiration
2040-11-10

AI Technical Summary

Technical Problem

Drones cannot accurately sense the location of surface objects in farmland, which makes them prone to hit obstacles and cause the consequences of machine damage and power outage.

Method used

The ground segmentation learning model training method is adopted to obtain sample point cloud data and standard ground surfaces, fit the initial sample ground plane, obtain sample distance distribution, and use this information to iterate the network learning model to obtain the ground segmentation learning model to identify ground plane.

Benefits of technology

It improves the accuracy and obstacle avoidance ability of the drone to identify the ground surface, reduces the impact of noise on the ground recognition, and enhances the safety of the drone.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112329946B_ABST
    Figure CN112329946B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention proposes a ground segmentation learning model training method, a ground plane determination method and related devices, wherein the ground segmentation learning model training method obtains sample point cloud data and the corresponding standard ground surface; fits the initial sample ground plane based on the sample point cloud data; obtains the sample distance distribution of the sample point cloud data relative to the initial sample ground plane; iterates the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model for identifying the ground plane. This solution takes into account the distance between the initial sample ground planes with more reliable distances, so that the trained segmentation model can more accurately identify the real ground surface from the input point cloud data, so that the drone can better avoid obstacles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a ground segmentation learning model training method, a ground plane determination method and related devices. Background Art

[0002] With the advancement of agricultural mechanization and intelligentization, more and more automated equipment is operating in farmland, and the recent application of drones in agriculture has also attracted attention. However, the complex environmental factors in farmland have seriously hindered the automation process of agriculture. Obstacles such as trees, windbreaks, electric poles and cable ties endanger the safety of operating equipment. Especially for operating equipment such as drones, which cannot accurately perceive the position of objects on the surface, it is easy for drones to collide with obstacles, resulting in the destruction of the machine and power outages. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a ground segmentation learning model training method, a ground plane determination method and related devices.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a ground segmentation learning model training method, the ground segmentation learning model training method comprising:

[0006] Obtain sample point cloud data and the corresponding standard ground surface;

[0007] Fitting an initial sample ground plane based on the sample point cloud data;

[0008] Obtaining a sample distance distribution of the sample point cloud data relative to the initial sample ground plane;

[0009] According to the sample distance distribution and the standard ground surface, a pre-selected network learning model is iterated to obtain a ground segmentation learning model for identifying the ground plane.

[0010] In an optional embodiment, the step of iterating the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain the ground segmentation learning model includes:

[0011] Inputting the sample distance distribution into the network learning model to obtain a sample output surface;

[0012] Obtaining a deviation evaluation value between the sample output surface and the standard surface;

[0013] The deviation evaluation value is used as a loss value, and the model parameters of the network learning model are adjusted by back propagation to obtain the ground segmentation learning model.

[0014] In an optional implementation, the ground segmentation learning model training method further includes: obtaining a sample color distribution corresponding to the sample point cloud data;

[0015] According to the sample distance distribution and the standard ground surface, the pre-selected network learning model is iterated to obtain a ground segmentation learning model, including:

[0016] According to the sample color distribution, the sample distance distribution and the standard ground surface, a pre-selected network learning model is iterated to obtain a ground segmentation learning model.

[0017] In a second aspect, an embodiment of the present invention provides a method for determining a ground plane, the method comprising:

[0018] Obtaining an initial ground plane based on three-dimensional point cloud data for representing a real-world scene;

[0019] Acquire the distance distribution of the three-dimensional point cloud data relative to the initial ground plane;

[0020] According to the distance distribution and the ground segmentation learning model, an optimal ground plane is obtained; wherein the ground segmentation learning model is a model trained by the ground segmentation learning model training method described in the above embodiment.

[0021] In an optional embodiment, the ground plane determination method also includes: obtaining the color distribution corresponding to the three-dimensional point cloud data; according to the distance distribution and the ground segmentation learning model, the step of obtaining the optimal ground plane includes: inputting the distance distribution and the color distribution into the ground segmentation learning model to obtain the optimal ground plane.

[0022] In an optional implementation, the step of obtaining the color distribution corresponding to the three-dimensional point cloud data includes:

[0023] Extracting color information corresponding to each point cloud point in the three-dimensional point cloud data;

[0024] According to the color information of all the point cloud points, the corresponding color distribution is constructed.

[0025] In an optional implementation, the step of obtaining the distance distribution of the three-dimensional point cloud data relative to the initial ground plane includes:

[0026] Calculating the distance value between each point in the point cloud and the initial ground plane;

[0027] The distance distribution is constructed based on all the distance values.

[0028] In an optional implementation, the method of obtaining the initial ground plane based on the three-dimensional point cloud data for representing the real-world scene includes:

[0029] Based on the three-dimensional point cloud data, the initial ground plane is fitted using a pre-selected algorithm.

[0030] In a third aspect, an embodiment of the present invention provides a ground segmentation learning model training device, the ground segmentation learning model training device comprising:

[0031] A generation module is used to obtain sample point cloud data and the corresponding standard ground surface;

[0032] A first fitting module, used for fitting an initial sample ground plane based on the sample point cloud data;

[0033] A first acquisition module, used for acquiring a sample distance distribution of the sample point cloud data relative to the initial sample ground plane;

[0034] The training module is used to iterate the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model for identifying the ground plane.

[0035] In a fourth aspect, an embodiment of the present invention provides a ground plane determining device, the ground plane determining device comprising:

[0036] A second fitting module is used to obtain an initial ground plane based on three-dimensional point cloud data used to represent a real-world scene;

[0037] A second acquisition module, used to acquire the distance distribution of the three-dimensional point cloud data relative to the initial ground plane;

[0038] A processing module is used to obtain an optimal ground plane according to the distance distribution and the ground segmentation learning model; wherein the ground segmentation learning model is a model trained using the ground segmentation learning model training method described in the above embodiment.

[0039] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the ground segmentation learning model training method described in the aforementioned embodiment; or the processor can execute the machine executable instructions to implement the ground plane determination method described in the aforementioned embodiment.

[0040] In an optional embodiment, the electronic device is a plant protection drone with an image acquisition module, and the plant protection drone acquires real images of real-world scenes through the image acquisition module to obtain the three-dimensional point cloud data.

[0041] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for training a ground segmentation learning model as described in the aforementioned embodiment is implemented; or when the computer program is executed by a processor, the method for determining a ground plane as described in any one of the aforementioned embodiments is implemented.

[0042] Compared with the prior art, the ground segmentation learning model training method provided by the embodiment of the present invention obtains sample point cloud data and the corresponding standard ground surface, and then fits the initial sample ground plane based on the sample point cloud data; obtains the sample distance distribution of the sample point cloud data relative to the initial sample ground plane, and uses the sample distance distribution and the standard ground surface to iterate the pre-selected network learning model to obtain a ground segmentation learning model for identifying the ground plane. That is, the distance information between the initial sample ground planes with relatively reliable point cloud distances is fully considered, so that the trained segmentation model can more accurately identify the ground surface from the input point cloud data, so as to more accurately separate the ground plane and the surface objects according to the ground surface, so that the drone can better avoid obstacles.

[0043] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of an electronic device provided by an embodiment of the present invention is shown.

[0046] Figure 2 A flowchart of the steps of a ground segmentation learning model training method provided by an embodiment of the present invention is shown.

[0047] Figure 3 A sub-step flow chart of step S103 provided in an embodiment of the present invention is shown.

[0048] Figure 4 A sub-step flow chart of step S104 provided in an embodiment of the present invention is shown.

[0049] Figure 5 A flow chart of the steps of a method for determining a ground plane provided by an embodiment of the present invention is shown.

[0050] Figure 6 A sub-step flow chart of step S202 provided in an embodiment of the present invention is shown.

[0051] Figure 7 A schematic diagram of a ground segmentation learning model training device provided by an embodiment of the present invention is shown.

[0052] Figure 8 A schematic diagram of a ground plane determination device provided by an embodiment of the present invention is shown.

[0053] Icon: 100-electronic device; 101-memory; 102-communication interface; 103-processor; 104-bus; 500-ground segmentation learning model training device; 501-generation module; 502-first fitting module; 503-first acquisition module; 504-training module; 600-ground plane determination device; 601-second fitting module; 602-second acquisition module; 603-processing module. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0056] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0057] The prerequisite for drone automatic flight operations is to identify obstacles in the flight space, such as trees, telephone poles, cable wires, etc.

[0058] In related technologies, in order to improve the obstacle avoidance capability of drones, the following methods are mainly used:

[0059] First, the real-time perception system directly senses the position of obstacles through onboard sensors. However, the accuracy of obstacle avoidance in this method depends on the sensitivity of the sensor. At the same time, this method lacks prejudgment, and it is easy to find that it is too late to adjust the flight posture when the obstacle is identified.

[0060] Second, aerial surveys generate DSMs and use high-level information to avoid obstacles. However, DSM reconstruction accuracy is not high, and the high-level information is simple, which is not suitable for accurate obstacle avoidance.

[0061] Third, the lidar point cloud uses RANSAC to fit the ground, separate the ground and surface objects, and cluster obstacles. This method has high accuracy, but its processing capability for point clouds with more noise is weak.

[0062] Based on the above, it can be seen that being able to accurately identify the ground surface is very important for drones to achieve accurate and efficient obstacle avoidance.

[0063] In order to improve the above-mentioned problems, realize accurate ground surface recognition, and thus improve the obstacle avoidance capability of the UAV, an embodiment of the present invention provides a ground segmentation learning model training method, a ground plane determination method, and related devices.

[0064] Please refer to Figure 1 , Figure 1 The block diagram of the electronic device 100 provided by the embodiment of the present invention is shown. The electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical device, and can also be a virtual machine arranged on a physical device, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine.

[0065] In addition, in some embodiments, the electronic device 100 is a drone. In another embodiment, the electronic device 100 may be a device that is communicatively connected to a drone.

[0066] In some embodiments, the electronic device may also be a plant protection drone with an image acquisition module, and the plant protection drone acquires real images of real-world scenes through the image acquisition module to obtain three-dimensional point cloud data.

[0067] The operating system of the electronic device 100 may be, but not limited to, Windows system, Linux system, etc. The electronic device 100 includes a memory 101, a communication interface 102, a processor 103 and a bus 104. The memory 101, the communication interface 102 and the processor 103 are connected via the bus 104. The processor 103 is used to execute executable modules stored in the memory 101, such as computer programs. Among them, the memory 101 may include a high-speed random access memory 101 (RAM: Random Access Memory), and may also include a non-volatile memory 101 (non-volatile memory), such as at least one disk storage 101. The communication connection between the electronic device 100 and the external device is realized through at least one communication interface 102 (which can be wired or wireless).

[0068] The bus 104 may be an ISA bus 104, a PCI bus 104, or an EISA bus 104, among others. Figure 1 Although only one bidirectional arrow is used in the diagram, it does not mean that there is only one bus 104 or only one type of bus 104 .

[0069] In some embodiments, the memory 101 is used to store programs, such as Figure 7 The ground segmentation learning model training device 500 shown. The ground segmentation learning model training device 500 includes at least one software function module that can be stored in the memory 101 in the form of software or firmware or fixed in the operating system (OS) of the electronic device 100. After receiving the execution instruction, the processor 103 executes the program to implement the ground segmentation learning model training method disclosed in the above embodiment of the present invention.

[0070] In some embodiments, the memory 101 is used to store programs, such as Figure 8 The ground plane determining device 600 shown. The ground plane determining device 600 includes at least one software function module that can be stored in the memory 101 in the form of software or firmware or fixed in the operating system (OS) of the electronic device 100. After receiving the execution instruction, the processor 103 executes the program to implement the ground plane determining method disclosed in the above embodiment of the present invention.

[0071] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 103 or instructions in the form of software. The above processor 103 can be a general processor 103, including a central processing unit 103 (CPU), a network processor 103 (NP), etc.; it can also be a digital signal processor 103 (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0072] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0073] First embodiment

[0074] Please refer to Figure 2 , Figure 2 FIG. 1 shows a ground segmentation learning model training method provided by an embodiment of the present invention. The ground segmentation learning model training method is applied to the electronic device 100. Figure 2 As shown, the above ground segmentation learning model training method may include the following steps:

[0075] Step S101, obtaining sample point cloud data and the corresponding standard ground surface.

[0076] The above sample point cloud data is a three-dimensional point cloud of a known and accurate surface. It can be understood that the above three-dimensional point cloud is a model used to simulate and present the real space, which is composed of a large number of points (i.e., point cloud points) with characteristic information (e.g., color information, position information). Therefore, the above sample point cloud data can be reconstructed based on the images collected in the real space. In order to obtain enough and diverse sample point cloud data, images of different real spaces can be collected separately for reconstruction to obtain multiple sample point cloud data.

[0077] The standard ground surface is composed of point cloud points representing the ground surface in real space in the sample point cloud data. In some embodiments, in response to a marking instruction input by a user, point cloud points belonging to the ground surface in real space can be determined from the sample point cloud data to form the standard ground surface.

[0078] Step S102: fitting an initial sample ground plane based on the sample point cloud data.

[0079] The above-mentioned initial sample ground plane is a ground plane obtained by calculation and fitting based on the sample point cloud data.

[0080] In some embodiments, an initial sample ground plane fitting may be performed based on the sample point cloud data by using a random sample consensus (RANSAC) algorithm or the like.

[0081] Step S103, obtaining the sample distance distribution of the sample point cloud data relative to the initial sample ground plane.

[0082] The above sample distance distribution is used to characterize the distance relationship between each point cloud point in the sample point cloud data and the initial sample ground plane.

[0083] Step S104, iterating the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model for identifying the ground plane.

[0084] The above network learning model can be, but is not limited to, a logistic regression model (LR), a support vector machine model (SVM) or a convolutional neural network model (CNN).

[0085] Among them, the above LR model is a classification model with simple and efficient algorithm implementation in machine learning, and its application scenarios are very wide. The above SVM model is one of the most robust and accurate methods among all well-known data mining algorithms. It belongs to a binary classification algorithm and can support linear and nonlinear classification. The above CNN model is a type of feedforward neural network (Feedforward Neural Networks) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform translation-invariant classification of input information according to their hierarchical structure.

[0086] Therefore, in some embodiments, those skilled in the art may select the type of network learning model and perform training according to actual needs.

[0087] It is understandable that when the ground segmentation learning model training method provided in the embodiment of the present invention performs model training, the sample distance distribution is used as a training factor. It is understandable that the above-mentioned sample distance distribution is used to characterize the distance relationship between the point cloud point and the initial sample ground plane. Although the initial sample ground plane is easily affected by noise, it can also relatively accurately characterize the real ground. Therefore, the above-mentioned sample distance distribution can be used to distinguish between the ground and non-ground to a certain extent. Using the sample distance distribution for model training can improve the ground recognition ability of the model obtained after training. Compared with directly using the fitted ground as the real ground, this method is more accurate and can effectively reduce the impact of noise points in the point cloud on the identification of the ground.

[0088] The implementation details of the embodiment of the present invention are described below:

[0089] In some embodiments, the above step S101 may be to use a drone to collect multiple groups of aerial images in different real spaces, wherein each group of aerial images corresponds to a real space. Three-dimensional reconstruction is performed based on each group of aerial images to obtain multiple sample point cloud data. Then, in response to the user's marking operation, multiple target point cloud points are determined from each sample point cloud data, and the point cloud area composed of the target point cloud points in each sample point cloud data is used as the corresponding standard ground surface.

[0090] In some embodiments, the step S102 may be to process the sample point cloud data using a RANSAC algorithm to obtain an expression for characterizing the ground surface, that is, an initial sample ground plane.

[0091] In some other embodiments, the step S102 may also be to process the sample point cloud data using a least squares algorithm to obtain an expression for characterizing the ground surface, that is, an initial sample ground plane.

[0092] Of course, other algorithms that can fit the approximate area of ​​the ground surface can also be used to fit the initial sample ground plane.

[0093] In some embodiments, Figure 3 As shown, the above step S103 may include the following steps:

[0094] Sub-step S103 - 1 , calculating the distance value between each point cloud point in the sample point cloud data and the initial sample ground plane.

[0095] In some embodiments, each point cloud point in the sample point cloud data is traversed to obtain a distance value between the traversed point cloud point and the initial sample ground plane.

[0096] Sub-step S103 - 2 , constructing a distance distribution according to all distance values ​​corresponding to the sample point cloud data.

[0097] In some embodiments, the distance distribution corresponding to the sample point cloud data may be constructed directly based on all distance values ​​corresponding to the sample point cloud data and the relative position relationship between each point cloud point.

[0098] In other embodiments, the distance distribution may be indirectly constructed based on all the distance values ​​corresponding to the sample point cloud data. For example, the distance distribution may be obtained by performing numerical distribution statistics based on all the corresponding distance values.

[0099] In some embodiments, the above-mentioned ground segmentation learning model training method also includes: obtaining a sample color distribution corresponding to the sample point cloud data.

[0100] In some embodiments, obtaining the sample color distribution corresponding to the sample point cloud data may include the following steps:

[0101] Sub-step S1, extracting the color information corresponding to each point cloud point in the sample point cloud data.

[0102] In some embodiments, each point cloud point in the sample point cloud data may be traversed to obtain the RGB value corresponding to each point cloud point.

[0103] Sub-step S2, constructing corresponding color distribution according to the color information of the point cloud points.

[0104] In some embodiments, the color distribution may be constructed directly based on the RGB values ​​of all the point cloud points in the sample point cloud data and the relative position relationship between the point cloud points.

[0105] In other embodiments, the color distribution can also be indirectly constructed based on the RGB values ​​of all the point cloud points in the sample point cloud data. For example, numerical distribution statistics can be performed on different color channels (R channel, G channel and B channel) based on the RGB values ​​of all the point cloud points, and a feature description vector can be constructed based on the numerical distribution statistics corresponding to the different color channels to serve as the color distribution.

[0106] On this basis, if Figure 4 As shown, the above step S104 may include the following steps:

[0107] Sub-step S104-1, inputting the sample color distribution and the sample distance distribution into the network learning model to obtain the sample output surface.

[0108] In some embodiments, the sample color distribution and the sample distance distribution may be fused and input into the network learning model. The output result after the network learning model processes the fused sample color distribution and sample distance distribution is used as the sample output surface.

[0109] In some embodiments, the method of fusing the sample color distribution and the sample distance distribution may be to splice the sample color distribution and the sample distance distribution.

[0110] Sub-step S104-2, obtaining a deviation evaluation value between the sample output surface and the standard surface.

[0111] In some embodiments, the ratio between the number of overlapping point cloud points in the sample output surface and the standard surface and the total number of point cloud points in the standard surface can be used as a deviation evaluation value.

[0112] Sub-step S104-3, using the deviation evaluation value as the loss value, back-propagating to adjust the model parameters of the network learning model to obtain a ground segmentation learning model.

[0113] In some embodiments, the network learning model can be iteratively trained according to the sample color distribution, sample distance distribution and standard ground surface corresponding to different sample point cloud data in accordance with the above sub-steps S104-1 to S104-3 until the iteration termination condition is triggered, and the model at this time is determined as the ground segmentation learning model. It can be understood that the above iteration termination condition can be a set number of iterations, or the obtained deviation evaluation value can meet the preset requirements.

[0114] It can be understood that when the ground segmentation learning model training method provided in this embodiment performs model training, it is considered that there is a color difference between the ground and non-ground in real space, which can be used to distinguish the ground and non-ground to a certain extent. Therefore, it is used as a sample factor for training to improve the ground recognition ability of the model obtained after training.

[0115] Second embodiment

[0116] Please refer to Figure 5 , Figure 5 FIG. 1 shows a method for determining a ground plane provided by an embodiment of the present invention. The above-mentioned method for determining a ground plane is applied to the above-mentioned electronic device 100. Figure 5 As shown, the above-mentioned ground plane determination method may include the following steps:

[0117] Step S201 : acquiring an initial ground plane based on three-dimensional point cloud data used to represent a real-world scene.

[0118] In some embodiments, the three-dimensional point cloud data can be obtained by three-dimensional reconstruction based on an aerial image of the space to be identified. The space to be identified is a real space (ie, a real-world scene) that requires ground plane identification.

[0119] In some embodiments, the above step S201 may be based on the three-dimensional point cloud data and use a pre-selected algorithm to fit the initial ground plane.

[0120] In some embodiments, the pre-selected algorithm may be a RANSAC algorithm. In other words, the step S201 may be based on the three-dimensional point cloud data and use the RANSAC algorithm to fit the initial ground plane.

[0121] In other embodiments, the pre-selected algorithm may be a least squares algorithm. In other words, the step S201 may be based on the three-dimensional point cloud data and use a least squares algorithm to fit the initial ground plane.

[0122] Step S202, obtaining the distance distribution of the three-dimensional point cloud data relative to the initial ground plane.

[0123] In some embodiments, Figure 6 As shown, the above step S202 may further include the following steps:

[0124] Sub-step S202-1, calculating the distance value between each point cloud point in the three-dimensional point cloud data and the initial ground plane.

[0125] In some embodiments, each point cloud point in the three-dimensional point cloud data is traversed to obtain a distance value between the traversed point cloud point and the initial ground plane.

[0126] Sub-step S202-2, constructing a distance distribution according to all distance values ​​corresponding to the three-dimensional point cloud data.

[0127] In some embodiments, the distance distribution corresponding to the three-dimensional point cloud data can be constructed directly based on all distance values ​​of the three-dimensional point cloud data and the relative position relationship between each point cloud point.

[0128] In other embodiments, the distance distribution may be indirectly constructed based on all distance values ​​of the three-dimensional point cloud data. For example, the numerical distribution statistics may be performed based on all distance values ​​to obtain the distance distribution corresponding to the three-dimensional point cloud data.

[0129] Step S203, obtaining the best ground plane according to the distance distribution and ground segmentation learning model.

[0130] In some embodiments, the distance distribution may be input into a ground segmentation learning model to obtain an optimal ground plane.

[0131] In some other embodiments, the above-mentioned ground plane determination method may further include the step of: obtaining a color distribution corresponding to the three-dimensional point cloud data.

[0132] In some embodiments, obtaining the color distribution corresponding to the three-dimensional point cloud data may include the following steps:

[0133] Sub-step S2-1, extracting the color information corresponding to each point cloud point in the three-dimensional point cloud data.

[0134] In some embodiments, each point cloud point in the three-dimensional point cloud data may be traversed to obtain the RGB value corresponding to each point cloud point.

[0135] Sub-step S2-2, constructing corresponding color distribution according to the color information of the point cloud points.

[0136] In some embodiments, the color distribution corresponding to the three-dimensional point cloud data can be constructed directly according to the RGB values ​​of all the point cloud points in the three-dimensional point cloud data and the relative position relationship between the point cloud points.

[0137] In other embodiments, the color distribution can also be indirectly constructed based on the RGB values ​​of all the point cloud points in the three-dimensional point cloud data. For example, numerical distribution statistics can be performed on different color channels (R channel, G channel and B channel) based on the RGB values ​​of all the point cloud points, and a feature description vector can be constructed based on the numerical distribution statistics corresponding to the different color channels to serve as the color distribution corresponding to the three-dimensional point cloud data.

[0138] On this basis, the above step S203 may be to input the distance distribution and the color distribution into the ground segmentation learning model to obtain the best ground plane. For example, the distance distribution and the color distribution are first fused, and the fused data is input into the ground segmentation learning model.

[0139] The above-mentioned ground segmentation learning model may be a model trained by using the ground segmentation learning model training method described in the first embodiment.

[0140] It can be seen that the ground plane determination method provided by the embodiment of the present invention obtains an accurate ground plane by utilizing the color difference between the ground and the sub-ground and the distance difference between each point cloud point in the three-dimensional point cloud data and the initial fitted ground plane, and performing regression learning with the ground segmentation learning model. This method can distinguish the ground plane and the non-ground plane from multiple angles, can cope with scenes with more noise points in the three-dimensional point cloud data, and improve the accuracy and stability of ground plane segmentation.

[0141] In order to execute the corresponding steps in the above embodiments and various possible methods, an implementation method of a ground segmentation learning model training device 500 is given below. Optionally, the ground segmentation learning model training device 500 can adopt the above Figure 1 The device structure of the electronic device 100 is shown. Figure 7 , Figure 7A functional module diagram of a ground segmentation learning model training device 500 provided in an embodiment of the present invention. It should be noted that the ground segmentation learning model training device 500 provided in this embodiment has the same basic principle and technical effects as those in the above-mentioned embodiments. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned embodiments. The ground segmentation learning model training device 500 includes: a generation module 501, a first fitting module 502, a first acquisition module 503 and a training module 504.

[0142] The generation module 501 is used to obtain sample point cloud data and the corresponding standard ground surface.

[0143] The first fitting module 502 is used to fit an initial sample ground plane based on the sample point cloud data.

[0144] The first acquisition module 503 is used to acquire the sample distance distribution of the sample point cloud data relative to the initial sample ground plane.

[0145] The training module 504 is used to iterate the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model for identifying the ground plane.

[0146] In some embodiments, the training module 504 is used to:

[0147] Inputting the sample color distribution and the sample distance distribution into the network learning model to obtain a sample output surface;

[0148] Obtaining a deviation evaluation value between the sample output surface and the standard surface;

[0149] The deviation evaluation value is used as a loss value, and the model parameters of the network learning model are adjusted by back propagation to obtain the ground segmentation learning model.

[0150] In some embodiments, the first acquisition module 503 in the above-mentioned ground segmentation learning model training model is also used to obtain the sample color distribution corresponding to the sample point cloud data.

[0151] The training module 504 is further used to iterate the pre-selected network learning model according to the sample color distribution, the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model.

[0152] In order to execute the corresponding steps in the above embodiments and various possible methods, an implementation method of a ground plane determination device 600 is given below. Optionally, the ground plane determination device 600 can adopt the above Figure 1 The device structure of the electronic device 100 is shown. Figure 8 , Figure 8 A functional module diagram of a ground plane determination device 600 provided in an embodiment of the present invention. It should be noted that the ground plane determination device 600 provided in this embodiment has the same basic principle and technical effects as those in the above-mentioned embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above-mentioned embodiment. The ground plane determination device 600 includes: a second fitting module 601, a second acquisition module 602 and a processing module 603.

[0153] The second fitting module 601 is used to obtain an initial ground plane based on three-dimensional point cloud data used to represent a real-world scene.

[0154] The second acquisition module 602 is used to acquire the color distribution corresponding to the three-dimensional point cloud data; and acquire the distance distribution of the three-dimensional point cloud data relative to the initial ground plane.

[0155] The processing module 603 is used to obtain the best ground plane according to the distance distribution and the ground segmentation learning model.

[0156] In some embodiments, the second acquisition module 602 is specifically used to:

[0157] Extracting color information corresponding to each point cloud point in the three-dimensional point cloud data;

[0158] According to the color information of all the point cloud points, the corresponding color distribution is constructed.

[0159] In some embodiments, the second acquisition module 602 is specifically used to:

[0160] Calculating the distance value between each point in the point cloud and the initial ground plane;

[0161] The distance distribution is constructed based on all the distance values.

[0162] In some embodiments, the second fitting module 601 is specifically used to fit an initial ground plane based on the three-dimensional point cloud data using a pre-selected algorithm.

[0163] In some embodiments, the processing module 603 is further used to: input the distance distribution and the color distribution into the ground segmentation learning model to obtain an optimal ground plane.

[0164] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 101 shown in the figure may be fixed in the operating system (OS) of the electronic device 100 and may be Figure 1The processor 103 in the memory is executed. Meanwhile, the data, program codes, etc. required to execute the above modules can be stored in the memory 101.

[0165] In summary, the embodiments of the present invention provide a ground segmentation learning model training method, a ground plane determination method and related devices, wherein the ground segmentation learning model training method obtains sample point cloud data and the corresponding standard ground surface; fits the initial sample ground plane based on the sample point cloud data; obtains the sample color distribution corresponding to the sample point cloud data; and obtains the sample distance distribution of the sample point cloud data relative to the initial sample ground plane; iterates the pre-selected network learning model according to the sample color distribution, the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model for identifying the ground plane. That is, the color information of the point cloud and the distance between the initial sample ground plane with a relatively reliable distance are fully considered, so that the trained segmentation model can more accurately identify the real ground surface from the input point cloud data, so that the drone can better avoid obstacles.

[0166] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0168] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A ground segmentation learning model training method, characterized in that: The ground segmentation learning model training method comprises: Acquire sample point cloud data and a corresponding standard ground surface; the sample point cloud data is a three-dimensional point cloud of a known accurate ground surface; Fitting an initial sample ground plane based on the sample point cloud data; Obtaining a sample distance distribution of the sample point cloud data relative to the initial sample ground plane; the sample distance distribution is used to characterize the distance relationship between each point cloud point in the sample point cloud data and the initial sample ground plane; According to the sample distance distribution and the standard ground surface, the pre-selected network learning model is iterated to obtain a ground segmentation learning model for identifying the ground plane.

2. The ground segmentation learning model training method according to claim 1, characterized in that: The step of iterating the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain the ground segmentation learning model comprises: Inputting the sample distance distribution into the network learning model to obtain a sample output surface; Obtaining a deviation evaluation value between the sample output surface and the standard surface; The deviation evaluation value is used as a loss value, and the model parameters of the network learning model are adjusted by back propagation to obtain the ground segmentation learning model.

3. The ground segmentation learning model training method according to claim 1, characterized in that: The ground segmentation learning model training method further includes: obtaining a sample color distribution corresponding to the sample point cloud data; According to the sample distance distribution and the standard ground surface, the pre-selected network learning model is iterated to obtain a ground segmentation learning model, including: According to the sample color distribution, the sample distance distribution and the standard ground surface, a pre-selected network learning model is iterated to obtain a ground segmentation learning model.

4. A method for determining a ground plane, characterized in that: The ground plane determination method comprises: Obtaining an initial ground plane based on three-dimensional point cloud data for representing a real-world scene; Acquire the distance distribution of the three-dimensional point cloud data relative to the initial ground plane; An optimal ground plane is obtained according to the distance distribution and the ground segmentation learning model; wherein the ground segmentation learning model is a model trained using the ground segmentation learning model training method described in any one of claims 1-3.

5. The method for determining ground plane according to claim 4, characterized in that: The method for determining the ground plane also includes: obtaining the color distribution corresponding to the three-dimensional point cloud data; and the step of obtaining the best ground plane according to the distance distribution and the ground segmentation learning model includes: inputting the distance distribution and the color distribution into the ground segmentation learning model to obtain the best ground plane.

6. The method for determining ground plane according to claim 5, characterized in that: The step of obtaining the color distribution corresponding to the three-dimensional point cloud data comprises: Extracting color information corresponding to each point cloud point in the three-dimensional point cloud data; According to the color information of all the point cloud points, the corresponding color distribution is constructed.

7. The method for determining ground plane according to claim 4, characterized in that: The step of obtaining the distance distribution of the three-dimensional point cloud data relative to the initial ground plane comprises: Calculating the distance between each point cloud point in the three-dimensional point cloud data and the initial ground plane; The distance distribution is constructed based on all the distance values.

8. The method for determining ground plane according to claim 4, characterized in that: The method of obtaining the initial ground plane based on the three-dimensional point cloud data used to represent the real-world scene includes: Based on the three-dimensional point cloud data, the initial ground plane is fitted using a pre-selected algorithm.

9. A ground segmentation learning model training device, characterized in that: The ground segmentation learning model training device comprises: A generation module, used to obtain sample point cloud data and a corresponding standard ground surface; the sample point cloud data is a three-dimensional point cloud of a known accurate ground surface; A first fitting module, used for fitting an initial sample ground plane based on the sample point cloud data; A first acquisition module is used to acquire a sample distance distribution of the sample point cloud data relative to the initial sample ground plane; the sample distance distribution is used to characterize the distance relationship between each point cloud point in the sample point cloud data and the initial sample ground plane; The training module is used to iterate the pre-selected network learning model according to the sample distance distribution and the standard ground surface to obtain a ground segmentation learning model for identifying the ground plane.

10. A ground plane determination device, characterized in that: The ground plane determining device comprises: A second fitting module is used to obtain an initial ground plane based on three-dimensional point cloud data used to represent a real-world scene; A second acquisition module, used to acquire the distance distribution of the three-dimensional point cloud data relative to the initial ground plane; A processing module is used to obtain an optimal ground plane according to the distance distribution and the ground segmentation learning model; wherein the ground segmentation learning model is trained using the ground segmentation learning model training method described in any one of claims 1-3.

11. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, the processor can execute the machine executable instructions to implement the ground segmentation learning model training method described in any one of claims 1-3; or the processor can execute the machine executable instructions to implement the ground plane determination method described in any one of claims 4-8.

12. The electronic device according to claim 11, characterized in that: The electronic device is a plant protection drone with an image acquisition module. The plant protection drone acquires real images of real-world scenes through the image acquisition module to obtain the three-dimensional point cloud data.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the ground segmentation learning model training method as described in any one of claims 1-3; or when the computer program is executed by a processor, it implements the ground plane determination method as described in any one of claims 4-8.

Citation Information

Patent Citations

  • Target recognition method and terminal, mobile device control method and terminal

    CN109255302A

  • Point cloud classification method and device based on deep learning, storage medium and equipment

    CN110232329A

  • Corn high-throughput phenotype measurement method and device based on harvester and harvester

    CN110969654A

  • Map ground segmentation method and device, computer equipment and storage medium

    CN111179274A

  • Ground segmentation method and device based on point cloud data

    CN111639682A