Point cloud data annotation method, device, storage medium and electronic device

Abstract

The present application proposes a point cloud data annotation method, device, storage medium and electronic device. First, the point cloud data to be annotated is used as the input of the semantic segmentation model to obtain a first annotation result, which includes the semantic recognition type of each point in the point cloud data; the target object in the orthophoto is obtained; the type of the target point is annotated as the target type, the target point is the point corresponding to the target object in the point cloud data to be annotated, and the target type is the type corresponding to the target object; the target object can be more accurately identified from the two-dimensional image data through two-dimensional image recognition technology, and based on the recognition result of the target object, the type of the point corresponding to the target object in the point cloud data to be annotated is annotated as the target type to obtain a second annotation result, and the first annotation result is merged with the second annotation result to quickly and accurately obtain the final point cloud annotation data, which significantly improves the efficiency of annotation.

Description

Technical Field

[0001] The present application relates to the field of control technology, and in particular to a point cloud data annotation method, device, storage medium and electronic device. Background Art

[0002] As the usage rate of various types of automated equipment in agriculture increases, more and more types of agricultural scene data can be obtained, providing necessary data for building three-dimensional map scenes. Building three-dimensional scene maps requires a large amount of annotated point cloud data, but a large amount of annotated point cloud data is difficult to obtain.

[0003] In the prior art, manual annotation methods are usually used to annotate point cloud data. The annotation tools cannot handle large point clouds and are prone to crash. In addition, the manual annotation method is inefficient and cannot generate a large amount of point cloud annotation information in a short time, and cannot fully annotate large-scale data. Summary of the invention

[0004] The purpose of the present application is to provide a point cloud data annotation method, device, storage medium and electronic device to at least partially improve the above-mentioned problems.

[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a point cloud data annotation method, the method comprising:

[0007] Using the point cloud data to be annotated as an input of a semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes a semantic recognition type of each point in the point cloud data;

[0008] Acquire a target object in an orthophoto, wherein the orthophoto is two-dimensional image data corresponding to the point cloud data;

[0009] Marking the type of the target point as the target type to obtain a second marking result, wherein the target point is a point in the point cloud data to be marked that corresponds to the target object, and the target type is a type corresponding to the target object;

[0010] The first annotation result and the second annotation result are fused to obtain final point cloud annotation data.

[0011] In a second aspect, an embodiment of the present application provides a point cloud data annotation device, the device comprising:

[0012] A processing unit, configured to use the point cloud data to be annotated as an input of a semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes a semantic recognition type of each point in the point cloud data;

[0013] The processing unit is also used to obtain a target object in an orthophoto, wherein the orthophoto is two-dimensional image data corresponding to the point cloud data;

[0014] The processing unit is further used to mark the type of the target point as the target type to obtain a second marking result, wherein the target point is a point in the point cloud data to be marked that corresponds to the target object, and the target type is a type corresponding to the target object;

[0015] A fusion unit is used to fuse the first annotation result with the second annotation result to obtain final point cloud annotation data.

[0016] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store one or more programs; when the one or more programs are executed by the processor, the above method is implemented.

[0018] Compared with the prior art, in a point cloud data annotation method, device, storage medium and electronic device provided by an embodiment of the present application, first, the point cloud data to be annotated is used as the input of the semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes the semantic recognition type of each point in the point cloud data; the target object in the orthophoto is obtained, wherein the orthophoto is the two-dimensional image data corresponding to the point cloud data; the type of the target point is annotated as the target type to obtain a second annotation result, wherein the target point is the point corresponding to the target object in the point cloud data to be annotated, and the target type is the type corresponding to the target object; the first annotation result is merged with the second annotation result to obtain the final point cloud annotation data. The target object can be more accurately identified from the two-dimensional image data through two-dimensional image recognition technology, and based on the recognition result of the target object, the type of the point corresponding to the target object in the point cloud data to be annotated is annotated as the target type to obtain the second annotation result, and the first annotation result is merged with the second annotation result to quickly and accurately obtain the final point cloud annotation data, which significantly improves the efficiency of annotation.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of a process for annotating point cloud data provided in an embodiment of the present application;

[0023] Figure 3 One of the flow charts of the point cloud data annotation method provided in the embodiment of the present application;

[0024] Figure 4 A schematic diagram of sub-steps of S102 provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of sub-steps of S103 provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the sub-steps of S101 provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of sub-steps of S101-4 provided in an embodiment of the present application;

[0028] Figure 8 One of the flow charts of the point cloud data annotation method provided in the embodiment of the present application;

[0029] Fig. 9 A schematic diagram of a unit of a point cloud data annotation device provided in an embodiment of the present application.

[0030] In the figure: 10 - processor; 11 - memory; 12 - bus; 13 - communication interface; 201 - processing unit; 202 - fusion unit. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0033] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0034] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0035] In the description of the present application, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0036] In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "disposed" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0037] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0038] The point cloud data in the embodiment of the present application refers to the scanning data recorded in the form of points, each point contains three-dimensional coordinates, and some may also contain color information, reflection intensity information, grayscale value, depth or number of returns and other information.

[0039] Building a three-dimensional scene map requires a large amount of annotated point cloud data, but a large amount of annotated point cloud data is difficult to obtain. After a lot of observations and summaries, the inventors found that computer vision has mature and stable development in the analysis technology of two-dimensional images. In agricultural scenes, fields and fruit trees account for the vast majority. These two types of data can be accurately identified through image technology. Therefore, a large amount of labeled point cloud data can be generated by combining three-dimensional point cloud with two-dimensional image technology. This overcomes the problem in the prior art that "manual annotation methods are usually used to annotate point cloud data, and annotation tools cannot carry large point clouds, which easily leads to crashes. In addition, the manual annotation method is inefficient, and it is impossible to generate a large amount of point cloud annotation information in a short time, and it is impossible to fully annotate large-scale data."

[0040] The present application embodiment provides an electronic device, which may be a computer, a server or a flight device. Figure 1 , a schematic diagram of the structure of an electronic device. The electronic device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12, and the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0041] The processor 10 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the point cloud data annotation method can be completed by the hardware integrated logic circuit in the processor 10 or the instructions in the form of software. The above-mentioned processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0042] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0043] The bus 12 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Figure 1 Although only one bidirectional arrow is used in the figure, it does not mean that there is only one bus 12 or only one type of bus 12 .

[0044] The memory 11 is used to store programs, such as programs corresponding to the point cloud data annotation device. The point cloud data annotation device includes at least one software function module that can be stored in the memory 11 in the form of software or firmware or fixed in the operating system (OS) of the electronic device. After receiving the execution instruction, the processor 10 executes the program to implement the point cloud data annotation method.

[0045] Possibly, the electronic device provided in the embodiment of the present application further includes a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus.

[0046] It should be understood that Figure 1 The structure shown is only a schematic diagram of a portion of the electronic device. The electronic device may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0047] The point cloud data annotation method provided in the embodiment of the present application can be applied to but not limited to Figure 1 For detailed procedures, please refer to the electronic equipment shown in Figure 2 :

[0048] S101: Use the point cloud data to be annotated as input to a semantic segmentation model to obtain a first annotation result.

[0049] The first annotation result includes the semantic recognition type of each point in the point cloud data.

[0050] Optionally, the semantic recognition type is, for example, a house, a telephone pole, a vehicle, or unrecognized (other), etc. When the semantic segmentation model determines that a certain point in the point cloud data belongs to a house, the semantic recognition type of the point is determined to be a house.

[0051] S102, obtaining a target object in the orthophoto.

[0052] Among them, the orthophoto is the two-dimensional image data corresponding to the point cloud data.

[0053] It should be noted that the orthophoto is a remote sensing image with orthographic projection properties. Optionally, the orthophoto is synthesized from the video corresponding to the point cloud data. The target object is an object of relatively mature two-dimensional image recognition technology, which can be the fields and fruit trees mentioned above.

[0054] As mentioned above, orthophotos are two-dimensional image data. After years of development, two-dimensional image recognition technology has achieved good results. Through two-dimensional image recognition technology, target objects such as fields and fruit trees can be identified from two-dimensional image data more accurately.

[0055] S106: Mark the type of the target point as the target type to obtain a second marking result.

[0056] The target point is a point in the point cloud data to be labeled that corresponds to the target object, and the target type is a type corresponding to the target object.

[0057] Optionally, based on the recognition result of the target object, the type of the point corresponding to the target object in the point cloud data to be annotated is annotated as the target type to obtain a second annotation result. The accuracy of the second annotation result is closely related to the accuracy of the two-dimensional image recognition technology, and the two-dimensional image recognition technology is relatively mature, so the accuracy of the second annotation result can be guaranteed.

[0058] S107: Fusing the first annotation result and the second annotation result to obtain final point cloud annotation data.

[0059] Optionally, after the target object is determined based on the two-dimensional image, the corresponding second annotation result is more accurate, and the first annotation result is fused with the second annotation result to improve the accuracy of the point cloud annotation data.

[0060] In summary, in the point cloud data annotation method provided in the embodiment of the present application, first, the point cloud data to be annotated is used as the input of the semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes the semantic recognition type of each point in the point cloud data; the target object in the orthophoto is obtained, wherein the orthophoto is the two-dimensional image data corresponding to the point cloud data; the type of the target point is annotated as the target type to obtain a second annotation result, wherein the target point is the point corresponding to the target object in the point cloud data to be annotated, and the target type is the type corresponding to the target object; the first annotation result is fused with the second annotation result to obtain the final point cloud annotation data. The target object can be more accurately identified from the two-dimensional image data through the two-dimensional image recognition technology, and based on the recognition result of the target object, the type of the point corresponding to the target object in the point cloud data to be annotated is annotated as the target type to obtain the second annotation result, and the two-dimensional image recognition technology is relatively mature, so the accuracy of the second annotation result can be guaranteed, and the first annotation result is fused with the second annotation result, thereby improving the accuracy of the point cloud annotation data.

[0061] exist Figure 2 On the basis of how to determine the target point, the embodiment of the present application also provides a possible implementation method, please refer to Figure 3 , point cloud data annotation methods also include:

[0062] S103, obtaining pixel coordinate contour boundary information of the target object in the orthophoto.

[0063] Optionally, the orthophoto has a matching pixel coordinate system, and each pixel point in the orthophoto has corresponding pixel coordinates. The pixel points corresponding to the edge contour of the target object constitute the pixel coordinate contour boundary information, that is, the pixel coordinate contour boundary information includes the pixel coordinates of all pixel points corresponding to the edge contour of the target object. It should be noted that the orthophoto does not need to be manually labeled.

[0064] S104, converting the pixel coordinate contour boundary information into three-dimensional boundary information in the world coordinate system.

[0065] The world coordinate system is a coordinate system corresponding to the point cloud data, and the world coordinate system may be the WGS84 world coordinate system.

[0066] Optionally, the pixel coordinate contour boundary information is converted into three-dimensional boundary information in the world coordinate system according to a preset mapping relationship. The mapping relationship is a conversion relationship from the pixel coordinate system to the world coordinate system. The three-dimensional boundary information includes the points corresponding to the edge contour of the target object in the point cloud data.

[0067] S105, determining the target point according to the coordinate area value screening method.

[0068] Among them, the target point is located within the three-dimensional boundary information.

[0069] Optionally, points located within the three-dimensional boundary information are selected as target points according to the coordinate region value selection method. The point cloud data in the three-dimensional space is selected by the coordinate boundary region value selection method to obtain point cloud data corresponding to the target object, such as point cloud data corresponding to fruit trees and fields.

[0070] It should be noted that the execution order of S101 and S102 - S106 is not limited, and S101 and S102 - S106 can be executed simultaneously.

[0071] exist Figure 3 Based on the content in S102, the present application embodiment also provides a possible implementation method, please refer to Figure 4 , S102 includes:

[0072] S102-1, obtaining tile data of the orthophoto.

[0073] Optionally, the tile data is a tile map pyramid model, which is a multi-resolution hierarchical model, where the resolution decreases from the bottom layer to the top layer of the tile pyramid, but the represented geographic range remains unchanged. The corresponding tile data can be obtained based on the orthophoto.

[0074] S102-2, obtaining the target object in the tile data.

[0075] Optionally, because tile data is a multi-resolution hierarchical model, obtaining the target object in the tile data requires less computational effort while ensuring the accuracy of obtaining the target object, compared to directly obtaining the target object in the orthophoto.

[0076] Please continue to refer to Figure 4 After obtaining the tile data of the orthophoto, regarding S103, the present application embodiment also provides a possible implementation method, please continue to refer to Figure 4 , S103 includes:

[0077] S103-1, obtaining pixel coordinate contour boundary information of the target object in the tile data.

[0078] Optionally, in the embodiment of the present application, the tile is an image of 256×256 pixels. The higher the tile level, the more detailed the map information represented. The tile coordinates are usually converted to the world coordinate system using the relationship of the Mercator projection.

[0079]

[0080]

[0081] Among them, (tileX, tileY) is the tile index, (pixelX, pixelY) is the specific pixel coordinate position of the image with the specified tile index; (lng, lat) is the longitude and latitude information in the world coordinate system.

[0082] exist Figure 3 On the basis of, when the target object is a fruit tree or a field, regarding the content in S103, the embodiment of the present application also provides a possible implementation method, please refer to Figure 5 , S103 also includes:

[0083] S103-2, obtaining the radius and center point coordinates of the fruit tree in the orthophoto according to the target detection algorithm.

[0084] Optionally, the target detection algorithm is used to detect the radius and center point coordinates of the fruit tree in the orthophoto. In a possible implementation, the target detection algorithm is also used to detect the radius and center point coordinates of the fruit tree in the tile data of the orthophoto.

[0085] S103-3, determining the pixel coordinate contour boundary information of the fruit tree based on the radius and center point coordinates of the fruit tree.

[0086] Optionally, with the center point as the center of the circle, the pixel coordinate contour boundary information of the fruit tree can be determined according to the radius.

[0087] S103-4, obtaining pixel coordinate contour boundary information of the field in the orthophoto according to the image segmentation algorithm.

[0088] In a possible implementation, the image segmentation algorithm is also used for the pixel coordinate contour boundary information of the field in the tile data of the orthophoto, that is, the pixel coordinates of all pixel points corresponding to the edge contour of the field.

[0089] exist Figure 2 Based on the content in S101, the present application embodiment also provides a possible implementation method, please refer to Figure 6 , S101 includes:

[0090] S101-1, using the point cloud data to be annotated as input to the semantic segmentation model to obtain the pre-identified type and corresponding confidence level of each point in the point cloud data.

[0091] For example, the pre-identified type of point A is house, and the corresponding confidence level is 90%.

[0092] S101 - 2 , determining the pre-recognition type of the first type point as a semantic recognition type.

[0093] The first type of point is a point whose confidence is greater than or equal to a first confidence threshold.

[0094] Optionally, the first confidence threshold is 75%. The pre-recognized type of point A is house, and the corresponding confidence is 90%, which is greater than 75%, so point A is a first type point, and the semantic recognition type of point A can be determined to be house.

[0095] S101-3, deleting the pre-identified type of the second type point.

[0096] The second type of point is a point whose confidence is less than the first confidence threshold.

[0097] For example, the pre-recognition type of point B is a pond, and the corresponding confidence is 60%, which is less than 75%, so point B is a second type point, and the pre-recognition type of point B is deleted.

[0098] S101 - 4 , determining the semantic recognition type of the second type of points according to the semantic recognition type of the first type of points in the search area, to obtain a first labeling result.

[0099] The search area is an area of ​​a preset size around the second type point. Continuing to refer to point B in the above example, all first type points whose distance from point B is less than the preset distance threshold are searched, and the search area is a range centered on point B and whose distance from point B is less than the preset distance threshold.

[0100] exist Figure 6 Based on the content in S101-4, the present application embodiment also provides a possible implementation method, please refer to Figure 7 , S101-4 includes:

[0101] S101-4-A, determine whether the number of first type points with the same semantic recognition type in the search area exceeds a second number threshold. If yes, execute S101-4-B, if no, execute S101-4-C.

[0102] Assuming that the second type point whose semantic recognition type is to be determined is point B, and the number of points whose semantic recognition type is house in the search area corresponding to point B is X, and X is greater than the second quantity threshold, S101-4-B can be executed to determine the semantic recognition type corresponding to point B as house. On the contrary, when the number of first type points with the same semantic recognition type in the search area does not exceed the second quantity threshold, S101-4-C will be executed.

[0103] S101-4-B, determining the semantic recognition type as the semantic recognition type of the second type point.

[0104] S101-4-C, mark the semantic recognition type of the second type of point as other.

[0105] Optionally, when there are multiple first-type points with the same semantic recognition type in the search area corresponding to point B, the number of which exceeds the second quantity threshold, for example, when the number of points with the semantic recognition type of house in the search area corresponding to point B is X, and when the number of points with the semantic recognition type of pond in the search area corresponding to point B is Y, X is greater than the second quantity threshold, and Y is greater than the second quantity threshold. The closest type is screened out based on the total number and average confidence of the first-type points with the same semantic recognition type, and the closest type is determined as the semantic recognition type of the second-type points. For example, if X is greater than Y, and the average confidence of the points with the semantic recognition type of house in the search area is greater than the average confidence of the points with the semantic recognition type of pond in the search area, the semantic recognition type of point B can be determined as house.

[0106] It should be noted that, in the embodiment of the present application, the types included in the first annotation result and the types included in the second annotation result may or may not be repeated. For example, when the target object corresponding to the second annotation result is a fruit tree and a field, the type included in the second annotation result is a fruit tree and a field. When the semantic segmentation model performs semantic segmentation on point cloud data, it may mark points suspected of being fruit trees and fields as other or unrecognized, or it may mark points suspected of being fruit trees and fields as fruit trees and fields.

[0107] When the semantic segmentation model performs semantic segmentation on point cloud data, points suspected to be fruit trees and fields can be marked as other or unidentified, which simplifies the model function and reduces the model's computational workload, thereby improving the computational efficiency of the semantic segmentation model.

[0108] In one possible implementation, a manual labeling method is used to obtain a small amount of point cloud data with labeling information of N categories. The N categories may not include the two categories of fruit trees and fields, or may include the two categories of fruit trees and fields. The labeled data is trained using a deep learning semantic segmentation method to obtain a corresponding semantic segmentation model.

[0109] The point cloud data in the embodiment of the present application can be processed by three-dimensional reconstruction technology corresponding to the video images obtained by the drone or other flying equipment to obtain point cloud data.

[0110] exist Figure 2 On the basis of how to optimize the semantic segmentation model, the present application embodiment also provides a possible implementation method, please refer to Figure 8 , point cloud data annotation methods also include:

[0111] S108, training a semantic segmentation model based on the point cloud annotation data.

[0112] Optionally, the semantic segmentation model is trained through point cloud annotation data to improve the accuracy of type recognition of the semantic segmentation model, so as to achieve the accuracy of target object recognition by two-dimensional image recognition technology and comprehensively improve the accuracy of type recognition of the semantic segmentation model.

[0113] See also Fig. 9 , Fig. 9 A point cloud data annotation device is provided in an embodiment of the present application. Optionally, the point cloud data annotation device is applied to the electronic device described above.

[0114] The point cloud data annotation device includes: a processing unit 201 and a fusion unit 202 .

[0115] The processing unit 201 is used to use the point cloud data to be annotated as an input of a semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes a semantic recognition type of each point in the point cloud data.

[0116] The processing unit 201 is further used to obtain the target object in the orthophoto, wherein the orthophoto is two-dimensional image data corresponding to the point cloud data.

[0117] The processing unit 201 is further configured to label the type of the target point as the target type to obtain a second labeling result, wherein the target point is a point corresponding to the target object in the point cloud data to be labeled, and the target type is a type corresponding to the target object.

[0118] Optionally, the processing unit 201 may execute the above-mentioned S101 , S102 and S106 .

[0119] The fusion unit 202 is used to fuse the first annotation result with the second annotation result to obtain final point cloud annotation data.

[0120] Optionally, the fusion unit 202 may execute the above S107.

[0121] The processing unit 201 is also used to obtain the pixel coordinate contour boundary information of the target object in the orthophoto; convert the pixel coordinate contour boundary information into three-dimensional boundary information in the world coordinate system, wherein the world coordinate system is the coordinate system corresponding to the point cloud data; determine the target point according to the coordinate area value screening method, wherein the target point is located within the three-dimensional boundary information. Optionally, the processing unit 201 can execute the above S103-S105.

[0122] The processing unit 201 is also used to obtain tile data of the orthophoto and obtain the target object in the tile data. Optionally, the processing unit 201 can execute the above S102-1 and S102-2.

[0123] The processing unit 201 is further configured to obtain pixel coordinate contour boundary information of the target object in the tile data. Optionally, the processing unit 201 may execute the above S1013-1.

[0124] When the target object is a fruit tree, the processing unit 201 is further used to obtain the radius and center point coordinates of the fruit tree in the orthophoto according to the target detection algorithm; and determine the pixel coordinate contour boundary information of the fruit tree according to the radius and center point coordinates of the fruit tree.

[0125] When the target object is a field, the processing unit 201 is further configured to obtain pixel coordinate contour boundary information of the field in the orthophoto according to an image segmentation algorithm.

[0126] Optionally, the processing unit 201 may execute the above-mentioned S103 - 2 to S103 - 4 .

[0127] The processing unit 201 is also used to use the point cloud data to be annotated as the input of the semantic segmentation model to obtain the pre-recognition type and corresponding confidence of each point in the point cloud data; determine the pre-recognition type of the first type of point as the semantic recognition type, wherein the first type of point is a point whose confidence is greater than or equal to the first confidence threshold; delete the pre-recognition type of the second type of point, wherein the second type of point is a point whose confidence is less than the first confidence threshold; determine the semantic recognition type of the second type of point based on the semantic recognition type of the first type of point in the search area, wherein the search area is an area of ​​a preset size around the second type of point. Optionally, the processing unit 201 can execute the above S101-1 to S101-4.

[0128] The processing unit 201 is further configured to determine whether the number of first-type points with the same semantic recognition type in the search area exceeds a second number threshold; if so, the semantic recognition type is determined to be the semantic recognition type of the second-type point. Optionally, the processing unit 201 may perform the above-mentioned S101-4-A to 101-4-B.

[0129] The processing unit 201 is further configured to train the semantic segmentation model according to the point cloud annotation data. Optionally, the processing unit 201 may execute the above-mentioned S108.

[0130] It should be noted that the point cloud data annotation device provided in this embodiment can execute the method flow shown in the above method flow embodiment to achieve the corresponding technical effect. For the sake of brief description, for parts not mentioned in this embodiment, please refer to the corresponding content in the above embodiment.

[0131] The present application also provides a computer-readable storage medium, which stores computer instructions and programs, and when the computer instructions and programs are read and run, the point cloud data annotation method of the above embodiment is executed. The storage medium may include a memory, a flash memory, a register, or a combination thereof.

[0132] An electronic device is provided below, which may be a computer device or a flight device. Figure 1 As shown, the above-mentioned point cloud data annotation method can be implemented; specifically, the electronic device includes: a processor 10, a memory 11, and a bus 12. The processor 10 can be a CPU. The memory 11 is used to store one or more programs. When the one or more programs are executed by the processor 10, the point cloud data annotation method of the above-mentioned embodiment is executed.

[0133] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0135] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0136] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0137] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A point cloud data annotation method, characterized in that: The method comprises: Using the point cloud data to be annotated as an input of a semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes a semantic recognition type of each point in the point cloud data; Acquire a target object in an orthophoto, wherein the orthophoto is two-dimensional image data corresponding to the point cloud data; Marking the type of the target point as the target type to obtain a second marking result, wherein the target point is a point in the point cloud data to be marked that corresponds to the target object, and the target type is a type corresponding to the target object; The first annotation result and the second annotation result are fused to obtain final point cloud annotation data, and the final point cloud annotation data is used to train the semantic segmentation model.

2. The point cloud data annotation method according to claim 1, characterized in that: After acquiring the target object in the orthophoto, the method comprises: Obtain the pixel coordinate contour boundary information of the target object in the orthophoto; Converting the pixel coordinate contour boundary information into three-dimensional boundary information in a world coordinate system, wherein the world coordinate system is a coordinate system corresponding to the point cloud data; The target point is determined according to a coordinate area value screening method, wherein the target point is located within the three-dimensional boundary information.

3. The point cloud data annotation method according to claim 2, characterized in that: The step of obtaining the target object in the orthophoto includes: Obtaining tile data of the orthophoto; Acquire a target object in the tile data; The step of obtaining pixel coordinate contour boundary information of the target object in the orthophoto includes: The step of obtaining pixel coordinate contour boundary information of the target object in the tile data.

4. The point cloud data annotation method according to claim 2, characterized in that: When the target object is a fruit tree, the step of obtaining pixel coordinate contour boundary information of the target object in the orthophoto includes: Obtaining the radius and center point coordinates of the fruit tree in the orthophoto according to the target detection algorithm; Determine pixel coordinate contour boundary information of the fruit tree according to the radius and center point coordinates of the fruit tree; When the target object is a field, the step of obtaining pixel coordinate contour boundary information of the target object in the orthophoto includes: The pixel coordinate contour boundary information of the field in the orthophoto is obtained based on the image segmentation algorithm.

5. The point cloud data annotation method according to claim 1, characterized in that: The step of using the point cloud data to be annotated as input of the semantic segmentation model to obtain a first annotation result includes: Using the point cloud data to be annotated as input to the semantic segmentation model, obtaining the pre-recognized type and corresponding confidence of each point in the point cloud data; Determining the pre-recognition type of the first type point as a semantic recognition type, wherein the first type point is a point whose confidence is greater than or equal to a first confidence threshold; Deleting the pre-identified type of the second type point, wherein the second type point is a point whose confidence is less than the first confidence threshold; The semantic recognition type of the second type of point is determined according to the semantic recognition type of the first type of point in a search area, wherein the search area is an area of ​​a preset size around the second type of point.

6. The point cloud data annotation method according to claim 5, characterized in that: The step of determining the semantic recognition type of the second type of point according to the semantic recognition type of the first type of point in the search area comprises: Determine whether the number of first-type points with the same semantic recognition type in the search area exceeds a second quantity threshold; If so, the semantic recognition type is determined as the semantic recognition type of the second type point.

7. A point cloud data annotation device, characterized in that: The device comprises: A processing unit, configured to use the point cloud data to be annotated as an input of a semantic segmentation model to obtain a first annotation result, wherein the first annotation result includes a semantic recognition type of each point in the point cloud data; The processing unit is also used to obtain a target object in an orthophoto, wherein the orthophoto is two-dimensional image data corresponding to the point cloud data; The processing unit is further used to mark the type of the target point as the target type to obtain a second marking result, wherein the target point is a point in the point cloud data to be marked that corresponds to the target object, and the target type is a type corresponding to the target object; A fusion unit is used to fuse the first annotation result with the second annotation result to obtain final point cloud annotation data, and the final point cloud annotation data is used to train the semantic segmentation model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store one or more programs; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

Images