Point cloud segmentation method, electronic device and system

Abstract

The invention discloses a point cloud segmentation method, an electronic device and a system. The method comprises the following steps: acquiring three-dimensional point cloud data; the three-dimensional point cloud data is clustered to obtain N sets, P sets exist in the N sets, the P sets meet the condition that different points in the same set correspond to different objects or points in different sets correspond to the same object, sparse annotations of each set are obtained, the sparse annotations comprise semantic annotations carried out on part of points in each set, and the semantic annotations comprise semantic annotations carried out on part of points in each set; the semantic annotation is used for indicating objects corresponding to part of points; and according to the sparse labels of the N sets, the three-dimensional point cloud data are clustered again to obtain M new sets, all points in each new set in the M new sets correspond to the same object, and points in different new sets correspond to different objects. According to the method, the set of the corresponding points of the object in the real environment in the three-dimensional point cloud data can be determined through sparse labeling of each set in the three-dimensional point cloud data, labeling of each point in the three-dimensional point cloud data is not needed, and labeling efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a point cloud segmentation method, electronic device and system. Background Art

[0002] Three-dimensional point cloud data (point cloud for short) is a collection of discrete points in a three-dimensional metric space, which is used to represent the spatial position of objects in the real environment. Three-dimensional point cloud data can be used to identify objects in the real environment. Taking the autonomous driving scenario as an example, the accurate perception of objects in the surrounding environment in the autonomous driving scenario is related to the safety and comfort of driving. Point cloud segmentation is the core task module of the autonomous driving perception system, which provides refined perception results of obstacles (such as cars, people, traffic cones, etc.) and drivable areas in automatic driving or parking scenarios, and can also provide rich semantic information for downstream tasks such as tracking and prediction.

[0003] At present, point cloud segmentation technology is mainly based on deep neural networks. Deep neural networks require a large amount of 3D point cloud data for training. Among them, the 3D point cloud data used for training is manually annotated point by point. For example, for a frame of point cloud, manual point-by-point annotation takes 1.5 to 4.5 hours. The annotation time cost is high, the efficiency is low, and it will affect the optimization iteration of the neural network. Summary of the invention

[0004] The embodiments of the present application provide a point cloud segmentation method, an electronic device, and a system for reducing labeling costs and improving efficiency.

[0005] In a first aspect, a point cloud segmentation method is provided, which can be applied to a first device. The first device can be, for example, a server. The method includes: acquiring three-dimensional point cloud data; clustering the three-dimensional point cloud data to obtain N sets, N is a positive integer, each of the N sets includes multiple points, there are P sets in the N sets, and the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object, and P is a positive integer less than or equal to N; acquiring sparse annotations for each set, the sparse annotations for each set include semantic annotations for some points in each set, and the semantic annotations are used to indicate the objects corresponding to some points; based on the sparse annotations of the N sets, re-clustering the three-dimensional point cloud data to obtain M new sets, M is a positive integer, each of the M new sets includes multiple points, all points in each new set correspond to the same object, and points in different new sets correspond to different objects.

[0006] In the embodiment of the present application, in the process of identifying an object through three-dimensional point cloud data, it is not necessary to manually annotate each point in the three-dimensional point cloud data, but only to semantically annotate some points in each of the N sets of three-dimensional point cloud data. Compared with "full annotation", the embodiment of the present application only needs to annotate some points in the three-dimensional point cloud data, which can reduce the time cost brought by annotation and improve annotation efficiency.

[0007] In one possible design, the N sets include a first set, all points in the first set correspond to a first object, and obtaining the sparse annotations of each set includes: obtaining the sparse annotations of the first set, the sparse annotations of the first set including a first semantic annotation for a first point in the first set, the first point is any point in the first set, and the first semantic annotation is used to indicate the first object.

[0008] That is to say, when all points in a set correspond to the same object, it is only necessary to perform the first semantic annotation on any point in the set, and there is no need to annotate every point in the set, thereby reducing the annotation time cost.

[0009] In one possible design, the P sets include a second set, different points in the second set correspond to different objects, and obtaining the sparse annotations of each set includes: obtaining the sparse annotations of the second set, the sparse annotations of the second set include a second semantic annotation for a second point in the second set and a third semantic annotation for a third point in the second set, the second point corresponds to a second object, the third point corresponds to a third object, the third object is different from the second object, the second semantic annotation is used to indicate the second object, and the third semantic annotation is used to indicate the third object.

[0010] That is to say, when different points in a set correspond to different objects, the different points in the set are semantically annotated respectively. For example, in the second set, point 1 corresponds to object 1, and point 2 corresponds to object 2. Then, point 1 and point 2 are semantically annotated respectively, and the semantic annotation of point 1 is used to indicate object 1, and the semantic annotation of point 2 is used to indicate object 2. Therefore, the sparse labeling of the set includes the semantic annotation of point 1 and the semantic annotation of point 2. It can be seen that the present application does not need to annotate each point in the set, which reduces the annotation time cost. Moreover, the present application annotates the different objects corresponding to different points in the set, which helps to re-cluster the three-dimensional point cloud data and improve the accuracy of object recognition.

[0011] In one possible design, the P sets include a third set and a fourth set, the fourth point in the third set and the fifth point in the fourth set correspond to the same object, and obtaining the sparse annotations of each set includes: obtaining the sparse annotations of the third set, the sparse annotations of the third set including a fourth semantic annotation for the fourth point; obtaining the sparse annotations of the fourth set, the sparse annotations of the fourth set including a fifth semantic annotation for the fifth point; wherein the fourth point and the fifth point both correspond to a fourth object, and the fourth semantic annotation and the fifth semantic annotation are both used to indicate the fourth object.

[0012] That is to say, when the points in one set and the points in another set correspond to the same object, the points in the one set and the points in the other set are labeled separately. Since the two points correspond to the same object, the semantic labels of the two points are the same. This helps to re-cluster the three-dimensional point cloud data and improve the accuracy of object recognition.

[0013] In one possible design, the three-dimensional point cloud data are re-clustered according to the sparse annotations of the N sets to obtain M new sets, including: semantically annotating each point of all points included in the three-dimensional point cloud data according to the sparse annotations of the N sets, the semantic annotation of each point being used to indicate the object corresponding to each point; and re-clustering the three-dimensional point cloud data according to the semantic annotations of all the points to obtain the M new sets.

[0014] In the present application, in the process of identifying objects through three-dimensional point cloud data, only sparse points in the three-dimensional point cloud data are manually annotated (for example, semantic annotation is performed on some points in each set of N sets), and the semantic annotation of each point in all points in the three-dimensional point cloud data can be determined by semantic annotation of sparse points. In simple terms, only some points in the three-dimensional point cloud data are manually annotated, and each point in the three-dimensional point cloud data is automatically annotated based on the manual annotation results of the said some points, so as to realize semi-automatic annotation of the three-dimensional point cloud data, that is, there is no need to completely rely on manual annotation, so as to improve the annotation efficiency.

[0015] In a possible design, the P sets include a fifth set, and the three-dimensional point cloud data are re-clustered according to the semantic annotations of all the points to obtain the M new sets, including: determining a target point in the P sets except the fifth set according to the semantic annotations of all the points, the semantic annotation of the target point being included in the sparse annotation of the fifth set, and a distance between the target point and the fifth set being less than a first distance; adding the target point to the fifth set to obtain a first new set, and the first new set is one of the M new sets.

[0016] Assuming that the real environment includes object A, the first device obtains three-dimensional point cloud data of the real environment. After clustering, the three-dimensional point cloud data obtains set 1 and set 2. The points in set 1 correspond to a part of object A, and the points in set 2 correspond to another part of object A. Obviously, this clustering is inaccurate. Therefore, the first device needs to re-cluster the three-dimensional point cloud data, for example, find the target point in set 2, and add the target point to set 1 to obtain a new set. The new set obtained after re-clustering is more accurate. Therefore, the present application can improve the accuracy of object recognition.

[0017] In one possible design, the P sets include a sixth set, and the three-dimensional point cloud data are re-clustered according to the semantic annotation of each point to obtain M new sets, including: according to the semantic annotation of each point, some points in the sixth set are moved out of the sixth set, wherein the sixth set after moving the some points out is a second new set, the semantic annotation of each point included in the second new set is the same, the semantic annotation of the some points is different from the semantic annotation of the points included in the second new set, and the second new set is one of the M new sets.

[0018] Assuming that the real environment includes object A and object B, the first device obtains three-dimensional point cloud data of the real environment. After clustering the three-dimensional point cloud data, set 1 is obtained, and some points in set 1 correspond to object A, and other points correspond to object B. Obviously, this clustering is inaccurate. Therefore, the first device needs to re-cluster the three-dimensional point cloud data, for example, remove the points corresponding to object B in set 1, and set 1 after removing the points corresponding to object B constitutes a new set, and the new set only corresponds to object A. The new set obtained after re-clustering is more accurate. Therefore, the present application can improve the accuracy of object recognition.

[0019] In a possible design, the N sets include a seventh set, and the semantic annotation of each point in all the points included in the three-dimensional point cloud data based on the sparse annotations of the N sets includes: when the sparse annotations of the seventh set include only one semantic annotation, determining that the semantic annotation of each point in all the points in the seventh set is the one semantic annotation; or, when the sparse annotations of the seventh set include a seventh semantic annotation and an eighth semantic annotation, determining that the semantic annotation of each point in a first part of the points in the seventh set is the seventh semantic annotation, and the semantic annotation of each point in a second part of the points is the eighth semantic annotation, wherein the seventh semantic annotation is used to indicate the object corresponding to the seventh point in the seventh set, the eighth semantic annotation is used to indicate the object corresponding to the eighth point in the seventh set, the distance between the first part of the points and the seventh point is less than the second distance, and the distance between the second part of the points and the eighth point is less than the third distance.

[0020] That is to say, in this application, only some points in the three-dimensional point cloud data are manually labeled. Based on the manual labeling results of the said some points, each point in the three-dimensional point cloud data is automatically labeled, thereby realizing semi-automatic labeling of the three-dimensional point cloud data, that is, there is no need to rely entirely on manual labeling, thereby improving labeling efficiency.

[0021] Exemplarily, the semantic annotation is used to indicate the object corresponding to the partial points, including: the semantic annotation includes the category of the object corresponding to the partial points. That is, in this application, the name and / or type of the object corresponding to the partial points in each set is marked to facilitate object recognition.

[0022] In a second aspect, a point cloud segmentation method is also provided. The method is applicable to a second device. The second device may be, for example, a terminal. The method includes: obtaining N sets of three-dimensional point cloud data, where N is a positive integer, each of the N sets includes multiple points, and there are P sets in the N sets, and the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object, and P is a positive integer less than or equal to N; displaying the N sets; receiving an input operation, the input operation is used to input a sparse annotation of each set, the sparse annotation of each set includes a semantic annotation of some points in each set, and the semantic annotation is used to indicate the objects corresponding to the some points; sending the sparse annotation of each set.

[0023] That is to say, after the terminal displays N sets of 3D point cloud data, the annotator semantically annotates some points in each of the N sets, and annotates the objects corresponding to the points. Therefore, in this application, it is not necessary to manually annotate each point in the 3D point cloud data, but only to semantically annotate the sparse points in the 3D point cloud data. Compared with "full annotation", it reduces the annotation time cost and improves the annotation efficiency.

[0024] In a possible design, different sets in the N sets are displayed differently.

[0025] It should be noted that, at present, the annotators manually annotate each point in the three-dimensional point cloud data, so the terminal displays the three-dimensional point cloud data, and the annotators can annotate point by point. In this application, the annotators annotate in sets, that is, semantically annotate some points in each set. Therefore, the terminal can display N sets to facilitate the operation of the annotators, and the different display methods of different sets can facilitate the annotators to distinguish different sets, reducing the difficulty of annotation for the annotators.

[0026] In a possible design, different sets in the N sets are displayed differently, including: points in different sets are displayed in different colors, and / or edge contours of different sets have different colors.

[0027] In the present application, the annotators annotate in units of sets. Therefore, the terminal can display N sets to facilitate the operation of the annotators. Moreover, the different display modes of different sets can facilitate the annotators to distinguish different sets, thereby reducing the difficulty of annotating for the annotators.

[0028] In one possible design, the N sets include a first set, which is one of the N sets except the P sets, and all points in the first set correspond to a first object. The receiving input operation includes: receiving a first operation, where the first operation is used to input a sparse annotation of the first set, and the sparse annotation of the first set includes a first semantic annotation of a first point in the first set, where the first point is any point in the first set, and the first semantic annotation is used to indicate the first object.

[0029] In one possible design, the P sets include a second set, different points in the second set correspond to different objects, and receiving an input operation includes: receiving a second operation, the second operation is used to input a sparse annotation of the second set, the sparse annotation of the second set includes a second semantic annotation for a second point in the second set and a third semantic annotation for a third point in the second set, the second point corresponds to a second object, the third point corresponds to a third object, the third object is different from the second object, the second semantic annotation is used to indicate the second object, and the third semantic annotation is used to indicate the third object.

[0030] In one possible design, the P sets include a third set and a fourth set, a fourth point in the third set and a fifth point in the fourth set correspond to the same object, and the receiving input operation includes: receiving a third operation, the third operation being used to input a sparse annotation of the third set, the sparse annotation of the third set including a fourth semantic annotation for the fourth point; receiving a fourth operation, the input operation being used to input a sparse annotation of the fourth set, the sparse annotation of the fourth set including a fifth semantic annotation for the fifth point; wherein the fourth point and the fifth point both correspond to a fourth object, and the fourth semantic annotation and the fifth semantic annotation are both used to indicate the fourth object.

[0031] In a third aspect, an electronic device is also provided, comprising a memory and one or more processors; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the device executes the method described in the first aspect or the second aspect above.

[0032] In a fourth aspect, a system is also provided, comprising:

[0033] a first device and a second device;

[0034] The first device is used to execute the method described in the first aspect above;

[0035] The second device is used to execute the method as described in the second aspect above.

[0036] In a fifth aspect, a computer-readable storage medium is also provided, comprising computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect or the second aspect above.

[0037] In a sixth aspect, a computer program product is also provided, comprising a computer program, which, when executed on a computer, enables the computer to execute the method described in the first or second aspect above.

[0038] In the seventh aspect, a chip is also provided, which is coupled to a memory in an electronic device and is used to call a computer program stored in the memory and execute the technical solution of the first or second aspect mentioned above. In the embodiment of the present application, "coupling" refers to the direct or indirect combination of two components with each other.

[0039] For the technical effects that can be achieved by the above-mentioned second to seventh aspects, please refer to the description of the technical effects that can be achieved by the corresponding design scheme in the above-mentioned first aspect, and this application will not repeat them here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application;

[0041] Figure 2 A schematic diagram of a process flow of a point cloud segmentation method provided in one embodiment of the present application;

[0042] FIG. 3A to FIG. 3B A schematic diagram of three-dimensional point cloud data provided by an embodiment of the present application;

[0043] Figure 4 A schematic diagram of N sets of three-dimensional point cloud data provided in an embodiment of the present application;

[0044] Figure 5A schematic diagram of semantically marking points in set 4 provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of semantically marking points in set 1 provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of semantically marking points in set 2 and set 3 provided in an embodiment of the present application;

[0047] Figure 8 A schematic diagram of M new sets of three-dimensional point cloud data provided by an embodiment of the present application;

[0048] Fig. 9 Another schematic diagram of M new sets of three-dimensional point cloud data provided by an embodiment of the present application;

[0049] Fig. 10A and Fig. 10B A schematic diagram of modifying a set provided in an embodiment of the present application;

[0050] Fig.11 Another schematic diagram of modifying a set provided in an embodiment of the present application;

[0051] Fig.12 Another schematic diagram of a flow chart of a point cloud segmentation method provided in an embodiment of the present application;

[0052] Fig.13 A schematic diagram of a system provided in accordance with an embodiment of the present application;

[0053] Fig.14 Another schematic diagram of a flow chart of a point cloud segmentation method provided in an embodiment of the present application;

[0054] Fig.15 A schematic diagram of point markings provided in an embodiment of the present application;

[0055] Fig.16 A schematic diagram of line segment annotation provided in an embodiment of the present application;

[0056] Fig.17 A schematic diagram of an electronic device provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0057] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0058] The at least one involved in the embodiments of the present application includes one or more; wherein, more than one means greater than or equal to two. In addition, it should be understood that in the description of this specification, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as an explicit or implicit relative importance, nor can they be understood as an explicit or implicit order. For example, the first device and the second device do not represent the importance of the two or the order of the two, but are only for distinguishing the description. In the embodiments of the present application, "and / or" is only a description of the association relationship, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship.

[0059] The directional terms mentioned in the embodiments of the present application, such as "up", "down", "left", "right", "inside", "outside", etc., are only references to the directions of the drawings. Therefore, the directional terms used are for better and clearer explanation and understanding of the embodiments of the present application, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0060] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the specification. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] Three-dimensional point cloud data, referred to as point cloud, is a collection of discrete points in a three-dimensional metric space, which is used to represent the spatial position of objects in a real environment. For example, three-dimensional point cloud data can be obtained through devices such as three-dimensional scanners, laser radar (LiDAR), and TOF cameras.

[0062] Three-dimensional point cloud data can be used to identify objects in a real environment. It should be understood that three-dimensional point cloud data includes a large number of discrete points. Identifying objects in a real environment through three-dimensional point cloud data can be understood as determining which points in the three-dimensional point cloud data correspond to object A and which points correspond to object B. Among them, identifying objects in a real environment through three-dimensional point cloud data can be achieved through point cloud segmentation technology. At present, point cloud segmentation technology can be based on deep neural networks. For example, three-dimensional point cloud data is input into a deep neural network, and the output result is calculated by the algorithm in the deep neural network. The output result is used to indicate which points in the three-dimensional point cloud data correspond to object A and which points correspond to object B. In order to improve accuracy, the deep neural network needs to be trained using three-dimensional point cloud data, and the three-dimensional point cloud data used for training needs to be manually annotated point by point, that is, each point in the three-dimensional point cloud data is manually annotated, which can be called "full annotation". Among them, annotating a point is to indicate the object corresponding to the point. The current "full annotation" method consumes a high time cost, is inefficient, and will affect the optimization iteration of the deep neural network.

[0063] In view of this, the present application provides a point cloud segmentation method, which is applicable to a first device, which may be, for example, a server. After acquiring three-dimensional point cloud data, the first device clusters the three-dimensional point cloud data to obtain N sets, where N is a positive integer, each of the N sets includes multiple points, and there are P sets in the N sets, and the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object, and P is a positive integer less than or equal to N. Afterwards, the first device acquires sparse annotations for each set, and the sparse annotations for each set include semantic annotations for some points in each set, and the semantic annotations are used to indicate the objects to which some points correspond. Based on the sparse annotations of the N sets, the first device re-clusters the three-dimensional point cloud data to obtain M new sets, where M is a positive integer, and each of the M new sets includes multiple points, all points in each new set correspond to the same object, and points in different new sets correspond to different objects. In the point cloud segmentation method provided in the embodiment of the present application, the three-dimensional point cloud data is clustered twice, and the accuracy of the N sets obtained by the first clustering is low, and the accuracy of the M new sets obtained by the second clustering is improved. Among them, the first clustering can use the point cloud clustering algorithm (which will be introduced later). Before the second clustering, some points in the three-dimensional point cloud data (for example, some points in each set of the N sets) need to be annotated. The second clustering can be performed based on the annotated results of some points to obtain more accurate M new sets. On the one hand, in the embodiment of the present application, it is not necessary to manually annotate each point in the three-dimensional point cloud data. Compared with "full annotation", the annotation time cost is reduced and the annotation efficiency is improved. On the other hand, the point cloud segmentation method provided in the embodiment of the present application has high accuracy. This is because, at present, point cloud segmentation is based on deep neural networks. If accurate segmentation results are to be obtained, a large amount of annotated three-dimensional point cloud data needs to be used to train the deep neural network, but the annotation cost is high (because full annotation is required). Otherwise, the deep neural network cannot obtain accurate segmentation results. Therefore, the point cloud segmentation method provided in the embodiment of the present application can improve the accuracy of point cloud segmentation on the basis of reducing the annotation cost.

[0064] The point cloud segmentation method provided in the embodiment of the present application can be applied to a variety of application scenarios, such as autonomous driving scenarios (e.g., unmanned driving scenarios), virtual reality (VR), augmented reality (AR), mixed reality (MR) and other game scenarios, etc.

[0065] The point cloud segmentation method provided in the embodiment of the present application is applicable to the system. The system includes a first device and a second device. The first device can be used for related processing of three-dimensional point cloud data (for example, clustering of three-dimensional point cloud data), and the second device can be used to display three-dimensional point cloud data to facilitate manual annotation by annotators. Among them, the first device can be a device with strong computing power, such as a server. The second device can be a device with a display function, such as a terminal. The terminal includes but is not limited to portable electronic devices such as tablet computers and laptops. The server can be various types of servers, such as cloud servers, application servers, etc., which are not limited in the embodiments of the present application.

[0066] Alternatively, the point cloud segmentation method provided in the embodiment of the present application may also be applicable only to the first device. For example, the first device has a certain computing capability and a display function. In this case, the first device can be used for related processing of three-dimensional point cloud data (for example, clustering of three-dimensional point cloud data), and can also be used to display three-dimensional point cloud data to facilitate manual annotation by annotators, that is, one device can also implement the point cloud segmentation method provided in the embodiment of the present application.

[0067] For ease of understanding, the point cloud segmentation method provided in the embodiment of the following text application is applicable to a system as an example, and the first device in the system is a server and the second device is a terminal as an example.

[0068] Figure 1 FIG. 1 shows a schematic diagram of the structure of an electronic device. The electronic device may be, for example, a terminal in a system. Figure 1 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0069] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a high-speed cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

[0070] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0071] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0072] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0073] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0074] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0075] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0076] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0077] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0078] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0079] The wireless communication function of the electronic device can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, antenna 1 can be reused as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0080] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0081] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to electronic devices. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0082] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology.

[0083] The display screen 194 is used to display the display interface of the application, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0084] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera 193.

[0085] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, and the software code of at least one application program, etc. The data storage area can store data (such as images, videos, etc.) generated during the use of the electronic device, etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory, etc.

[0086] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as pictures and videos are stored in the external memory card.

[0087] The electronic device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0088] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The receiver 170B, also known as the "earpiece", can be one or more, and is used to convert audio electrical signals into sound signals.

[0089] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. Headphone jack 170D is used to connect wired headphones. Pressure sensor 180A is used to sense pressure signals and convert pressure signals into electrical signals. Gyroscope sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (i.e., x, y, and z axes) can be determined by gyroscope sensor 180B. Gyroscope sensor 180B can be used for anti-shake shooting.

[0090] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation. The magnetic sensor 180D includes a Hall sensor. The electronic device can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes). The distance sensor 180F is used to measure the distance. The electronic device can measure the distance through infrared or laser. The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect the temperature. The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the human vocal part. The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The electronic device can receive a key input and generate a key signal input related to the user settings and function control of the electronic device. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 195, or pulled out from the SIM card interface 195 to achieve contact and separation with the electronic device.

[0091] Understandably, Figure 1 The components shown do not constitute a specific limitation on the terminal. The terminal in the embodiment of the present invention may include Figure 1 In addition, Figure 1 The combination / connection relationship between the components can also be adjusted and modified.

[0092] For example, see Figure 2 , is a flow chart of a point cloud segmentation method provided in an embodiment of the present application. The method is applicable to a system, which includes a server and a terminal. Figure 2 As shown, the process includes:

[0093] S201, the server obtains three-dimensional point cloud data.

[0094] Taking the autonomous driving scenario as an example, assume that a laser radar is installed on the vehicle, which can be used to collect 3D point cloud data and send the collected 3D point cloud data to the server. For easy understanding, please refer to Figure 3A , which is a schematic diagram of three-dimensional point cloud data. The three-dimensional point cloud data includes discrete points, each of which is represented by a black solid point.

[0095] S202, the server clusters the three-dimensional point cloud data to obtain N sets, where N is a positive integer, each of the N sets includes multiple points, there are P sets in the N sets, and the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object, and P is a positive integer less than or equal to N.

[0096] In some embodiments, the server may use a point cloud clustering algorithm to cluster the three-dimensional point cloud data to obtain N sets. The point cloud clustering algorithm includes but is not limited to a density clustering algorithm (Density Based Spatial Clustering of Applications with Noise, DBSCAN), a hierarchical density clustering algorithm (Hierarchical Density Based Spatial Clustering of Applications with Noise, HDBSCAN), a hybrid density clustering algorithm (HDBSCAN+DBSCAN), etc. The embodiments of this application do not describe in detail the DBSCAN, HDBSCAN and other algorithms. Alternatively, the server may also use other methods to cluster the three-dimensional point cloud data to obtain N sets. In short, the embodiments of this application do not limit the implementation method of S202.

[0097] Optionally, considering that in most cases in autonomous driving scenarios, the 3D point cloud data collected comes from street scenes, so basically each collected 3D point cloud data contains the point cloud corresponding to the ground. For example, see Figure 3B, the points corresponding to the ground in the three-dimensional point cloud data are represented by hollow points. Therefore, in some other embodiments, the server may first perform ground detection on the three-dimensional point cloud data, and after detecting the points corresponding to the ground, cluster the remaining points (such as points corresponding to pedestrians, vehicles, etc.) (for example, using a point cloud clustering algorithm to determine N sets of the remaining points), which can improve the clustering efficiency. Among them, the ground may include roads, sidewalks, lawns, etc., which are usually extended in the horizontal direction and can be used as planes for load-bearing objects. Among them, ground detection can use any ground detection algorithm, such as the Random Sample Consensus (RANSAC) algorithm. Taking the RANSAC algorithm as an example, the algorithm can fit a plane from the three-dimensional point cloud data and determine that the plane is the ground. There is a possibility that when using the RANSAC algorithm for ground detection, multiple planes may be fitted, and the server can determine the ground from multiple planes, for example, through a test method (such as a plane slope check algorithm, a plane center of gravity height check algorithm, etc.), the ground is identified from the multiple fitted planes. Optionally, considering the diversity of ground categories (for example, ground categories include roads, sidewalks, lawns, etc.) and the diversity of scenes (for example, urban roads, mountain roads, rural roads, etc.), the three-dimensional point cloud data can be first divided into multiple sub-point cloud sets, and then ground detection is performed on each sub-point cloud set. Taking the RANSAC algorithm as an example, the algorithm is used to perform a ground detection once for each sub-point cloud set to identify the ground in the sub-point cloud set. In this way, each sub-point cloud set contains 1 or 0 ground point clusters, and multiple ground point clusters constitute the point cloud corresponding to the ground.

[0098] Optionally, the N sets include a foreground set and / or a background set. The points included in the foreground set correspond to foreground objects in the real environment, such as cars and people. The points included in the background set correspond to background objects in the real environment, such as walls and the ground.

[0099] Optionally, each of the N sets obtained in S202 may have an edge contour. The shape of the edge contour of each set is not limited in this application. It is understandable that each of the N sets having an edge contour can facilitate the labeling personnel to distinguish different sets and facilitate the labeling of points in the set. The specific process will be described later. Figure 3B Taking the three-dimensional point cloud data shown in FIG. 1 as an example, clustering the non-ground corresponding points in the three-dimensional point cloud data is performed to obtain four sets, including set 1 to set 4, such as Figure 4 As shown, each of the four sets has an edge contour.

[0100] As mentioned above, identifying objects through three-dimensional point cloud data means determining which points in the three-dimensional point cloud data correspond to object A and which points correspond to object B. Among them, all points corresponding to object A belong to one set, and all points corresponding to object B belong to another set. In these two sets, all points in the same set correspond to the same object, and points in different sets correspond to different objects. Therefore, the accuracy of the N sets obtained in S202 is relatively low. Because there are P sets in the N sets, and the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object. The P sets include at least one of the following:

[0101] (1), P sets include set 1. Different points in set 1 correspond to different objects. For example, some points in set 1 correspond to object A, and other points correspond to object B. For example, see Figure 4 , some points in set 1 correspond to cars, and the other points correspond to people.

[0102] (2), the P sets include set 2 and set 3. The points in set 2 and the points in set 3 correspond to the same object. Figure 4 For example, the points in set 2 correspond to the front of the tanker truck, and the points in set 3 correspond to the body of the tanker truck.

[0103] Therefore, the accuracy of the N sets obtained in S202 is low. In order to improve the accuracy, the embodiment of the present application will obtain sparse annotations for each of the N sets, and re-cluster the three-dimensional point cloud data according to the sparse annotations of each set. The specific implementation process will be described later.

[0104] It is understandable that, in addition to the P sets, there are other sets among the N sets, such as Figure 4 For example, set 4 is also included. All points in set 4 correspond to the same object (eg, a car), and the points in set 4 correspond to different objects than the points in other sets.

[0105] S203: The server sends N sets to the terminal.

[0106] S204: The terminal displays N sets.

[0107] S205 , the terminal receives an input operation, where the input operation is used to input a sparse annotation of each set, where the sparse annotation of each set includes a semantic annotation of some points in each set, where the semantic annotation is used to indicate an object corresponding to the some points.

[0108] Case 1: The N sets include the first set, and all points in the first set correspond to the same object. For example, the first set is Figure 4Set 4, Set 2, Set 3, etc. in the first set. In this case, S205 includes: the terminal receives a first operation for the first set, and the first operation is used to input a sparse labeling of the first set. The sparse labeling of the first set includes a semantic labeling of the first point in the first set, and the semantic labeling is used to indicate the object, for example, the semantic labeling includes the category of the object. In this case, since all points in the first set correspond to the same object, the sparse labeling of the first set includes only one semantic labeling, and the semantic labeling is used to indicate the object. The first point is any point in the first set. Figure 4 Take set 4 in the example, the annotator semantically annotates a point in set 4 as "car". The sparse annotations of set 4 include: car, see Figure 5 That is to say, for the first set, the annotator only needs to annotate any point in the set, which saves the annotation time cost.

[0109] Case 2: The N sets include the second set, and different points in the second set correspond to different objects. For example, the first part of the points in the second set corresponds to object A, and the second part of the points corresponds to object B. For example, the second set can be Figure 4 Set 1 in the data set. In this case, S205 includes: the terminal receives a second operation for the second set, and the second operation is used to input sparse annotations of the second set. The sparse annotations of the second set include: semantic annotations of the second point in the first part of points and semantic annotations of the third point in the second part of points. Among them, the second point is any point in the first part of points. The semantic annotation of the second point is used to indicate object A, for example, including the category of object A. The third point is any point in the second part of points. The semantic annotation of the third point is used to indicate object B, for example, including the category of object B. That is, the sparse annotations of the second set include two semantic annotations, and the two semantic annotations are used to indicate different objects. Figure 4 Take set 1 in the example, some points in set 1 correspond to cars, and some points correspond to people. The annotator annotates the second point in set 1 as: car, and the third point as: person. Figure 6 As shown. Therefore, the sparse labels of set 1 include: car and person. That is to say, for the second set, the labeler only needs to label the two points in the set, saving the labeling time cost.

[0110] Case 3: The N sets include the third set and the fourth set, and the fourth point in the third set and the fifth point in the fourth set correspond to the same object. For example, the third set is Figure 4 The fourth set is Figure 4Set 3 in the description. In this case, S205 includes: the terminal receives a third operation for the third set, and the third operation is used to input the sparse annotations of the third set. The sparse annotations of the third set include: a semantic annotation performed on the fourth point in the third set, and the semantic annotation is used to indicate the object corresponding to the fourth point, for example, the semantic annotation includes the category of the object corresponding to the fourth point. S205 also includes: the terminal receives a fourth operation for the fourth set, and the fourth operation is used to input the sparse annotations of the fourth set. The sparse annotations of the fourth set include: a semantic annotation performed on the fifth point in the fourth set, and the semantic annotation is used to indicate the object corresponding to the fifth point, for example, the semantic annotation includes the category of the object corresponding to the fifth point. The semantic annotation of the fourth point is the same as the semantic annotation of the fifth point, because the fourth point and the fifth point correspond to the same object. For example, see Figure 7 , Set 2 and Set 3 correspond to the same object. The semantic label of the fourth point in Set 2 is car, and the semantic label of the fifth point in Set 3 is car.

[0111] S206: The terminal sends the sparse annotation of each of the N sets to the server.

[0112] S207, the server re-clusters the three-dimensional point cloud data according to the sparse annotations of the N sets to obtain M new sets, where M is a positive integer, each of the M new sets includes multiple points, all points in each new set correspond to the same object, and points in different new sets correspond to different objects.

[0113] In some embodiments, S207 may include step 1 and step 2. Step 1: semantically annotate each point in all points included in the three-dimensional point cloud data according to the sparse annotations of the N sets, and the semantic annotation of each point is used to indicate the object corresponding to each point. Step 2: re-cluster the three-dimensional point cloud data according to the semantic annotations of all points to obtain the M new sets.

[0114] The following describes the implementation process of step 1.

[0115] As mentioned above, a sparse annotation of a set may include only one semantic annotation (for example, case 1 in the above text), or may include multiple semantic annotations, and different semantic annotation pairs indicate different objects (for example, case 2 in the above text). In different cases, the implementation method of step 1 is different.

[0116] For case 1, that is, when the sparse annotations of a set contain only one semantic annotation, it is determined that the semantic annotation of each point in all points in the set is the one semantic annotation. Figure 7 Take cluster 4 in the example, the sparse annotation of cluster 4 only includes one semantic annotation: car, so the semantic annotations of all points in cluster 4 are: car.

[0117] For the second case, that is, the sparse annotation of the set includes semantic annotation 1 and semantic annotation 2, wherein semantic annotation 1 is used to indicate object A corresponding to point 1 in the set, and semantic annotation 2 is used to indicate object B corresponding to point 2 in the set, and object A is different from object B. In this case, one way is to determine that the semantic annotation of each point in all the points in the set is semantic annotation 3, and semantic annotation 3 is used to indicate non-object C, and object C is all objects except object A and object B in all objects corresponding to the three-dimensional point cloud data. Another way is to determine that the semantic annotation of each point in all the remaining points in the set except point 1 and point 2 is semantic annotation 3, and semantic annotation 3 is used to indicate non-object C, the semantic annotation of point 1 is semantic annotation 1, and the semantic annotation of point 2 is semantic annotation 2. Among them, object C is all objects except object A and object B in all objects corresponding to the three-dimensional point cloud data. Continue with Figure 7 Take set 1 in the example, the sparse annotations of set 1 include: people and cars, where people are the semantic annotations of point 1 and cars are the semantic annotations of point 2. Then, the semantic annotations of other points in the set except point 1 and point 2 are all non-tree (assuming Figure 7 The objects corresponding to the three-dimensional point cloud data shown include people, cars and trees, and no other objects). Another way is to determine the semantic annotation of each point in the first part of the points in the set as semantic annotation 1, and the semantic annotation of each point in the second part of the points as semantic annotation 2. The distance between the first part of the points and point 1 is less than the second distance, and the distance between the second part of the points and point 2 is less than the third distance.

[0118] It can be seen from the above step 1 that in the embodiment of the present application, the semantic annotation of each point is not manually annotated point by point, but is first manually annotated in units of sets to obtain sparse annotations for each set, and then each point in the three-dimensional point cloud data is automatically annotated point by point based on the sparse annotations of each set, thereby realizing semi-automatic annotation of three-dimensional point cloud data.

[0119] The implementation process of step 2 is described below.

[0120] Optionally, the server may determine Q sets belonging to the same category among the N sets, count the average size of the Q sets, and cluster the Q sets again based on the average size of the Q sets using a point cloud clustering algorithm (e.g., a density clustering algorithm) to obtain multiple new sets. Counting the average size of the Q sets may include: finding the two farthest points in each of the Q sets in a bird's-eye view, the distance between the two points being the size of each set, that is, obtaining the size of each set in the Q sets, and then calculating the average size of the Q sets.

[0121] Two examples are listed below to demonstrate the effects of the embodiments of the present application.

[0122] For example, if a point in a set corresponds to the same object as a point in another set, the server expands to the neighborhood with the set as the center according to the semantic annotations of all points, and searches for a target point in other sets, the semantic annotation of the target point is included in the sparse annotation of the set, and the distance between the target point and the set is less than the first distance, and the target point is added to the set to obtain a new set, which is one of the M new sets. Figure 7 Take set 3 in as an example. The sparse labels of set 3 include car. With set 3 as the center, expand to the neighborhood and find target point 1 (target point 1 is in set 2). The semantic label of target point 1 is car and the distance with set 3 is less than the first distance. Therefore, aggregate target point 1 into set 3 to obtain a new set, such as Figure 8 Please compare Figure 8 and Figure 7 , Figure 7 Originally, a car was divided into two sets (for example, set 2 and set 3), with low accuracy. Using the technical solution of the present application, the same car belongs to the same set (for example, set 6).

[0123] For easier understanding, see Fig. 9 (a), the real environment includes a tanker truck. After the first clustering of the 3D point cloud data, set A includes a few points, which correspond to a small part of the tanker truck. The accuracy of set A is low. After clustering again (for example, neighborhood expansion), set A is adjusted to set B, as shown in Fig. 9 (b), set B includes more points, which correspond to most of the tanker truck. Therefore, the accuracy of the set obtained by the point cloud segmentation method provided in the embodiment of the present application is improved.

[0124] For another example, if different points in a set correspond to the same object, then according to the semantic annotation of each point, some points in the set are removed from the set, wherein the set after the removed points is a new set, and the semantic annotation of each point included in the new set is the same, and the semantic annotation of the removed points is different from the semantic annotation of the points included in the new set. Figure 7 For example, some points in set 1 correspond to cars, and some points correspond to people. Therefore, the server removes the points in set 1 that are semantically labeled as people, and retains the points that are semantically labeled as cars. The remaining points form a new set, for example Figure 8 7. Optionally, the removed points form another new set, e.g. Figure 8 Please compare Figure 8 and Figure 7 , Figure 7In the example, two different objects are clustered into one set (for example, set 1), with low accuracy. Using the technical solution of the present application, the two different objects belong to two different sets (for example, set 7 and set 8).

[0125] It should be noted that although Fig. 9 In the figure, set 7 and set 8 seem to have an intersection, but because the 3D point cloud data is distributed in the 3D space, that is, each point has depth information, set 7 and set 8 actually have no intersection.

[0126] Optionally, each of the M new sets may have an edge contour. For example, after clustering the point cloud again using a point cloud clustering algorithm (such as a density clustering algorithm), the edge contour of each new set may be automatically outlined without manual outlining.

[0127] Optionally, after the server re-clustered the 3D point cloud data to obtain M new sets, it may also modify one or more of the M new sets. The modification method includes but is not limited to at least one of the following:

[0128] Correction method 1: The server sends the first image and M new sets of three-dimensional point cloud data to the terminal. The first image is an image corresponding to the three-dimensional point cloud data, which can be a three-dimensional image or a two-dimensional image. For example, a laser radar and a camera are provided on the vehicle, the laser radar is used to collect three-dimensional point cloud data, and the camera is used to collect the first image. The three-dimensional point cloud data and the first image are different forms of data corresponding to the real environment (for example, the same street view) in which the vehicle is currently located. The terminal displays the first image and displays the three-dimensional point cloud data on the first image, which can be understood as mapping the three-dimensional point cloud data to the first image. The M new sets in the three-dimensional point cloud data are displayed in one-to-one correspondence with the M objects on the first image. For example, the first image includes object A, and the M new sets include new set A. All points included in the new set A correspond to object A, so the new set A is displayed at the location of object A on the first image. It can be understood that the points in the new set A cover the upper layer of the area where object A is located on the first image. The annotator determines whether the new set A matches the object A in the first image. Here, "matching" can include: whether the area enclosed by the edge contour of the new set A is larger or smaller than the area where the object A in the first image is located. For example, see Fig. 10A (a), the first image includes object A (e.g., a vehicle), and the new set A (points in the area enclosed by the solid line) is displayed on the upper layer of object A, but the area enclosed by the edge contour of the new set A is smaller than the area where object A is located in the first image. In this case, the annotator can adjust the new set A. For example, the annotator draws a two-dimensional box outside the object (e.g., the circumscribed rectangle of object A), such as Fig. 10A(b), the area of ​​the two-dimensional box is larger than the area of ​​the area enclosed by the edge contour of the new set A. The terminal assigns the same semantic annotation to all points in the two-dimensional box, which is used to indicate object A, that is, all points in the two-dimensional box belong to the same object A (i.e., the car). Therefore, the edge contour of the new set A is adjusted to Fig. 10A (c). Obviously, through the correction operation of the annotator, the original Fig. 10A The new set A of (a) is modified to Fig. 10A (c) The new set A after correction is more accurate. For the convenience of prompting, the terminal can display all points in the two-dimensional box with the same color, that is, Fig. 10A In (c), all points within the edge contour are displayed in the same color. Fig. 10A In (a), the points inside the edge contour of the new set A are displayed in the first color, and the points outside the edge contour are displayed in the second color. After the correction operation by the annotator, Fig. 10A In (c), all points in the new set display the same color, for example, the first color or the third color.

[0129] For another example, since instances of the barrier class are often close together, it is easy to label these instances as the same instance. Fig. 10B (a), the real environment includes trash cans and seats. Since the trash cans and seats are close to each other and their shapes are not very different, after using the technical solution of this application, the trash cans and seats may be divided into the same set, such as set A. Therefore, Fig. 10B In (a), the points corresponding to the trash can and the seat are displayed in the same way, such as showing the same color or Fig. 10B (a) Use "----------" to indicate. The annotator can make corrections, such as Fig. 10B (b), the labeler uses two correction boxes to frame the trash can and the seat respectively. Among them, correction box 1 corresponds to the trash can, and correction box 2 corresponds to the seat. The points in the same annotation box are considered to be the same set. Therefore, set A is corrected into two sets, namely set B (including the points in correction box 1) and set C (including the points in correction box 2). The points in the same set are displayed in the same way. For example, the points in set B are displayed in one color, and the points in set C are displayed in another color. Or, Fig. 10B (c), points in set B use Indicates that the points in set C are represented by “…”. Therefore, through this correction method, the accuracy of point cloud segmentation can be improved.

[0130] Correction method 2: Considering that 3D point cloud data is a discrete point in a 3D space, the distribution of points is different under different viewing angles (such as front view, left view, right view, top view, top view, etc.). Take the main view as an example, and take one of the M new sets as an example, the new set corresponds to object A. Therefore, from the main view, all points corresponding to object A are already in a new set. When the view is moved (such as the left view), it is found that there are still some points that are not in the new set, but these points also correspond to object A. For example, see Fig.11 (a) In the main perspective, the new set already includes all the points of object A (e.g., a person). When the view is changed to the top-down perspective, some points (points corresponding to object A) are not included in the new set, such as Fig.11 (b) It should be noted that when the viewing angle changes, the edge contour of the new set will also change due to the need for coordinate transformation. Fig.11 (a) and Fig.11 (b) The edge contour of the new set A is different. Therefore, in order to ensure accuracy, in the present application, after the server re-clustering the three-dimensional point cloud data to obtain M new sets, the M new sets can be corrected by continuously adjusting different viewing angles. Taking one of the new sets as an example, the server determines that the first annotation box of the new set in the first view (for example, the main view) surrounds all points of object A. The server maps the center of the first annotation box to the second view (for example, the left view) through coordinate transformation to obtain a second annotation box; if the second annotation box does not contain all points of the object A in the second view, the size of the second annotation box is adjusted to obtain a third annotation box so that all points of the object A in the second view are included in the third annotation box. The server maps the center of the third annotation box to the first view through coordinate transformation to obtain a fourth annotation box. If the fourth annotation box does not contain all the points of the object A in the first view, the size of the fourth annotation box is adjusted and the adjusted annotation box is mapped to the second view to check whether it contains all the points of the object A in the second view, until a annotation box is determined that can contain all the points of the object A in the first view and all the points of the object A in the second view after coordinate transformation. Fig.11 (a) In the main view, the first annotation box of object A contains all the points of object A in the main view, and then the first annotation box is mapped to the second annotation box in the top view through coordinate transformation, such as Fig.11 (b) If the second annotation box does not contain all the points of object A in the top view, adjust the size of the second annotation box to obtain a third annotation box, such as Fig.11 (c) The third annotation box contains all the points of object A in the top view, and then the center of the third annotation box is mapped to the main view through coordinate transformation to obtain the fourth annotation box, such as Fig.11(d), the fourth annotation box contains all the points of object A in the main view. Fig.11 The adjustment process adjusts the first annotation box to the fourth annotation box, and the points in the fourth annotation box are used as a new set with higher accuracy because the fourth annotation box contains all points of object A in the main view and also contains all points of object A in the top view.

[0131] In summary, see Fig.12 , is another flow chart of the point cloud segmentation method provided in the embodiment of the present application. Fig.12 , the process includes: 1. Input 3D point cloud data. Figure 1 S201 in .2. Ground detection. For ground detection, please refer to the previous description. 3. Point cloud clustering. Corresponding Figure 1 S202 in .4. Manual sparse annotation. Figure 1 S203-S206 in .5. Automatic semantic annotation. Figure 1 Step 1 in S207: According to the sparse annotations of the N sets, semantic annotation is performed on each point in all points included in the three-dimensional point cloud data, and the semantic annotation of each point is used to indicate the object corresponding to each point. 6. Automatic instance annotation. Figure 1 Step 2 in S207 of : re-clustering the 3D point cloud data according to the semantic annotations of all points to obtain the M new sets. 7. Instance correction. The correction process is described above and will not be repeated here.

[0132] In the above embodiments, the technical solution provided by the present application is implemented by a system as an example, wherein the system includes a server and a terminal. In other embodiments, the system may also include a database, a server and a terminal device, such as Fig.13 . The database can be integrated in the server or terminal, or it can be independent of the server and the terminal. The database is used to store 3D point cloud data. For example, the vehicle stores the collected 3D point cloud data in the database. The server includes a point cloud segmentation module and a point cloud annotation module. The point cloud segmentation module is used to cluster the 3D point cloud data. The point cloud annotation module is used to semantically annotate the points in the 3D point cloud data. The terminal includes an artificial sparse annotation module. The specific actions to be performed by each module will be described later. Fig.14 In the description.

[0133] See also Fig.14 , is another flow chart of the point cloud segmentation method provided in one embodiment of the present application. This flow chart can be applied to Fig.13 The system shown. Fig.14 The process includes:

[0134] S1401, the point cloud segmentation module in the server obtains three-dimensional point cloud data from the database.

[0135] For example, the vehicle stores the collected 3D point cloud data in a database, and the 3D point cloud data is 3D point cloud data that has not been segmented. Therefore, the point cloud segmentation module in the server in S1401 can obtain the 3D point cloud data from the database.

[0136] S1402, the point cloud segmentation module in the server clusters the three-dimensional point cloud data to obtain N sets.

[0137] Among them, the implementation principle of S1402 is the same as Figure 2 The same as S202 in, for the sake of brevity of the specification, it will not be repeated here.

[0138] S1403, the point cloud segmentation module in the server sends N sets to the terminal.

[0139] S1404: The terminal displays N sets.

[0140] Optionally, in order to facilitate the annotator to distinguish different sets, when the terminal displays N sets, different sets may be displayed in different ways. The different display modes of different sets include at least one of the following: different display colors and / or shapes of points in different sets; displaying the edge contours of each set and the colors and / or shapes of the edge contours of different sets are different. For example, the terminal displays N sets, for example Figure 4 , which will not be repeated here.

[0141] S1405, the terminal receives an input operation, the input operation is used to input a sparse label for each of the N sets, the sparse label for each set includes a semantic label for a portion of the points in each set, the semantic label is used to indicate an object corresponding to the portion of the points.

[0142] The following takes one of the N sets as an example, and takes the example that all points in the set correspond to one object to illustrate the labeling process of the set.

[0143] Optionally, the labeling methods include point labeling and line segment labeling. Point labeling can be understood as the labeler selecting a point in the set and semantically labeling the point, and the semantic labeling is used to indicate the object corresponding to the point, such as the category of the object. Line segment labeling can be understood as the labeler drawing a line segment and semantically labeling all the sets that the line segment passes through, and the semantic labeling is used to indicate the objects corresponding to the points in all the sets that the line segment passes through, such as the category of the object. In other words, line segment labeling can realize cross-set labeling. For example, two sets correspond to the same object, such as the ground. Then, the labeler does not need to label the two sets separately, but only needs to draw a line segment that passes through the two sets, and then label the line segment as the ground, so the sparse labeling of the two sets is the ground.

[0144] Take point annotation as an example, see Fig.15 , the three-dimensional point cloud data includes multiple sets. Taking set A as an example, the annotator can annotate point 1 in set A, for example, as a car. Optionally, in the interactive interface with the annotator, the terminal can render the color of the points in each set in real time according to the current manual sparse annotation situation to prompt the annotator of the current point cloud annotation situation. For example, after the annotator has finished annotating set A, according to the category of set A, all points in set A are rendered as the color corresponding to the category agreed in advance. For example, if it is agreed in advance that vehicles correspond to orange, then the points in set A will be displayed in orange.

[0145] Take line segment annotation as an example, see Fig.16 , the annotator draws one or more line segments, and the set of 3D point cloud data whose distance to the line segment is less than a threshold will be given the same semantic annotation, that is, corresponding to the same category, such as the ground.

[0146] It is understandable that in addition to the two annotation methods of "point annotation" and "line segment annotation", there may be other annotation methods, for example, the annotator may also annotate by drawing circles, etc. In short, in this application, the annotator can use a user-friendly (i.e., convenient for the annotator to operate) annotation method to semantically annotate the points in the three-dimensional point cloud data, which is convenient to operate and has a good experience; and the manual annotation is performed in units of sets, not point by point, which reduces the complexity of annotation and improves the efficiency of annotation.

[0147] S1406: The terminal sends the sparse annotations of N sets to the server.

[0148] S1407, the point cloud annotation module in the server re-clusters the three-dimensional point cloud data according to the sparse annotations of the N sets to obtain M new sets, each of the M new sets includes multiple points, all points in each new set correspond to the same object, and points in different new sets correspond to different objects.

[0149] Among them, the implementation principle of S1407 is the same as Figure 2 The implementation principle of S207 is the same as that of the embodiment, and will not be repeated here for the sake of brevity.

[0150] S1408, the point cloud annotation module in the server sends M new sets of three-dimensional point cloud data to the database.

[0151] Optionally, before S1408, the M new sets may be modified, and the server sends the modified M new sets to the database. The modification method is described above and will not be repeated here.

[0152] For ease of understanding, the currently used "full annotation" method and the point cloud segmentation method provided by this application are compared through experimental testing. The comparison results are shown in Table 1 below:

[0153] Table 1

[0154]

[0155] It can be seen from Table 1 above that if the "full annotation" method is used, the manual annotation speed is about 90 minutes / frame, that is, it takes 90 minutes to manually annotate the three-dimensional point cloud data corresponding to a frame of image. If the point cloud segmentation method of the present application (i.e. sparse annotation) is used, the annotation speed is 14 minutes / frame, that is, it takes 14 minutes to manually annotate the three-dimensional point cloud data corresponding to a frame of image. Obviously, the "full annotation" method can only annotate 16% of the data in the same amount of time. Therefore, the annotation method of the present application is more efficient. In addition, sparse annotation is used to train semantic segmentation tasks and panoramic segmentation tasks, and the indicators that can be obtained are 72.8mIoU and 62.6PQ. The indicators obtained by "full annotation" for training the model with 16% of the fully annotated data are 71.4mIoU and 59.2PQ, respectively, which are worse than the results of the sparse annotation model (the higher the indicator value, the better).

[0156] Fig.17 The electronic device 1700 provided in the embodiment of the present application is a schematic diagram of the structure. The electronic device 1700 may be the server or terminal mentioned above. Fig.17 As shown, the electronic device 1700 may include: one or more processors 1701; one or more memories 1702; a communication interface 1703, and one or more computer programs 1704, and the above-mentioned components may be connected via one or more communication buses 1705. The one or more computer programs 1704 are stored in the above-mentioned memories 1702 and are configured to be executed by the one or more processors 1701, and the one or more computer programs 1704 include instructions. For example, when the electronic device 1700 is the server mentioned above, the instructions may be used to execute the relevant steps of the server in the above corresponding embodiments, for example, Figure 2 or Fig.14 When the electronic device 1700 is the terminal mentioned above, the instruction can be used to execute the steps of the terminal in the above corresponding embodiment, for example, Figure 2 or Fig.14 The communication interface 1703 is used to implement the communication between the electronic device 1700 and other devices, for example, the communication interface may be a transceiver.

[0157] In the embodiments provided by the present application, the method provided by the embodiments of the present application is introduced from the perspective of the electronic device as the execution subject. In order to implement the functions in the methods provided by the embodiments of the present application, the electronic device may include a hardware structure and / or a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a function of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0158] As used in the above embodiments, the term "when..." or "after..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrase "when determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)", depending on the context. In addition, in the above embodiments, relational terms such as first and second are used to distinguish one entity from another, without limiting any actual relationship and order between the entities.

[0159] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0160] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc. In the absence of conflict, the solutions of the above embodiments can be used in combination.

[0161] Note: A portion of this patent application document contains material which is subject to copyright protection. The copyright owner reserves all rights reserved except for the production of copies of the material in the patent file or patent record in the Patent Office.

Claims

1. A point cloud segmentation method, characterized in that: The method comprises: Obtain 3D point cloud data; Clustering the three-dimensional point cloud data to obtain N sets, where N is a positive integer, each of the N sets includes a plurality of points, and the N sets include P sets, where the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object, and P is a positive integer less than or equal to N; Acquire sparse annotations of each set, wherein the sparse annotations of each set include semantic annotations performed on some points in each set, and the semantic annotations of some points are used to indicate objects corresponding to the some points; According to the sparse annotations of the N sets, the three-dimensional point cloud data are re-clustered to obtain M new sets, where M is a positive integer, each of the M new sets includes multiple points, all points in each new set correspond to the same object, and points in different new sets correspond to different objects.

2. The method according to claim 1, characterized in that The N sets include a first set, all points in the first set correspond to a first object, and obtaining sparse annotations of each set includes: Sparse annotations of the first set are obtained, where the sparse annotations of the first set include a first semantic annotation performed on a first point in the first set, where the first point is any point in the first set, and the first semantic annotation is used to indicate the first object.

3. The method according to claim 1 or 2, characterized in that: The P sets include a second set, different points in the second set correspond to different objects, and obtaining the sparse annotations of each set includes: Obtain sparse annotations of the second set, the sparse annotations of the second set including a second semantic annotation for a second point in the second set and a third semantic annotation for a third point in the second set, the second point corresponds to a second object, the third point corresponds to a third object, the third object is different from the second object, the second semantic annotation is used to indicate the second object, and the third semantic annotation is used to indicate the third object.

4. The method according to any one of claims 1 to 3, characterized in that: The P sets include a third set and a fourth set, a fourth point in the third set and a fifth point in the fourth set correspond to the same object, and obtaining the sparse annotations of each set includes: Acquire the sparse annotations of the third set, wherein the sparse annotations of the third set include a fourth semantic annotation performed on the fourth point; Acquire the sparse annotations of the fourth set, wherein the sparse annotations of the fourth set include a fifth semantic annotation performed on the fifth point; The fourth point and the fifth point both correspond to a fourth object, and the fourth semantic annotation and the fifth semantic annotation are both used to indicate the fourth object.

5. The method according to any one of claims 1 to 4, characterized in that: The three-dimensional point cloud data is re-clustered according to the sparse annotations of the N sets to obtain M new sets, including: According to the sparse annotations of the N sets, semantically annotating each point among all the points included in the three-dimensional point cloud data, wherein the semantic annotation of each point is used to indicate an object corresponding to each point; The three-dimensional point cloud data is re-clustered according to the semantic annotations of all the points to obtain the M new sets.

6. The method according to claim 5, characterized in that The P sets include a fifth set, and the three-dimensional point cloud data is re-clustered according to the semantic annotations of all the points to obtain the M new sets, including: Determine a target point in the other sets of the P sets except the fifth set according to the semantic annotations of all the points, the semantic annotation of the target point is included in the sparse annotation of the fifth set, and the distance between the target point and the fifth set is less than the first distance; The target point is added to the fifth set to obtain a first new set, where the first new set is one of the M new sets.

7. The method according to claim 5 or 6, characterized in that: The P sets include a sixth set, and the three-dimensional point cloud data is re-clustered according to the semantic annotation of each point to obtain M new sets, including: According to the semantic labeling of each point, some points in the sixth set are moved out of the sixth set, wherein the sixth set after moving out some points is a second new set, the semantic labeling of each point in the second new set is the same, the semantic labels of the moved some points are different from the semantic labels of the points in the second new set, and the second new set is one of the M new sets.

8. The method according to any one of claims 5 to 7, characterized in that: The N sets include a seventh set, and the semantic annotation of each point in all points included in the three-dimensional point cloud data according to the sparse annotations of the N sets includes: When the sparse annotations of the seventh set include only one semantic annotation, determining that the semantic annotation of each point in all points in the seventh set is the one semantic annotation; or When the sparse annotations of the seventh set include seventh semantic annotations and eighth semantic annotations, determine that the semantic annotation of each point in the first part of points in the seventh set is the seventh semantic annotation, and the semantic annotation of each point in the second part of points is the eighth semantic annotation, wherein the seventh semantic annotation is used to indicate the object corresponding to the seventh point in the seventh set, the eighth semantic annotation is used to indicate the object corresponding to the eighth point in the seventh set, the distance between the first part of points and the seventh point is less than the second distance, and the distance between the second part of points and the eighth point is less than the third distance.

9. The method according to any one of claims 1 to 8, characterized in that: The semantic annotation of the partial points is used to indicate the objects corresponding to the partial points, including: The semantic annotation of the partial points includes the category of the object corresponding to the partial points.

10. A point cloud segmentation method, characterized in that: The method comprises: Acquire N sets of three-dimensional point cloud data, where N is a positive integer, each of the N sets includes a plurality of points, and there are P sets among the N sets, and the P sets satisfy: different points in the same set correspond to different objects, or points in different sets correspond to the same object, and P is a positive integer less than or equal to N; Display the N sets; receiving an input operation, the input operation being used to input a sparse annotation of each set, the sparse annotation of each set comprising a semantic annotation performed on some points in each set, the semantic annotation being used to indicate an object corresponding to the some points; The sparse annotations for each set are sent.

11. The method according to claim 10, characterized in that Different sets in the N sets are displayed in different ways.

12. The method according to claim 11, characterized in that Different sets in the N sets are displayed in different ways, including: points in different sets are displayed in different colors, and / or edge contours of different sets are displayed in different colors.

13. The method according to any one of claims 10 to 12, characterized in that: The N sets include a first set, the first set is one of the N sets except the P sets, all points in the first set correspond to a first object, and the receiving input operation includes: A first operation is received, where the first operation is used to input sparse annotations of the first set, where the sparse annotations of the first set include performing a first semantic annotation on a first point in the first set, where the first point is any point in the first set, and where the first semantic annotation is used to indicate the first object.

14. The method according to any one of claims 10 to 13, characterized in that: The P sets include a second set, different points in the second set correspond to different objects, and receiving an input operation includes: A second operation is received, where the second operation is used to input sparse annotations of the second set, where the sparse annotations of the second set include a second semantic annotation for a second point in the second set and a third semantic annotation for a third point in the second set, where the second point corresponds to a second object, the third point corresponds to a third object, the third object is different from the second object, the second semantic annotation is used to indicate the second object, and the third semantic annotation is used to indicate the third object.

15. The method according to any one of claims 10 to 14, characterized in that: The P sets include a third set and a fourth set, a fourth point in the third set and a fifth point in the fourth set correspond to the same object, and the receiving input operation includes: receiving a third operation, the third operation being used to input the sparse annotations of the third set, the sparse annotations of the third set comprising a fourth semantic annotation performed on the fourth point; receiving a fourth operation, wherein the input operation is used to input the sparse annotations of the fourth set, and the sparse annotations of the fourth set include a fifth semantic annotation performed on the fifth point; The fourth point and the fifth point both correspond to a fourth object, and the fourth semantic annotation and the fifth semantic annotation are both used to indicate the fourth object.

16. An electronic device, characterized in that: The device comprises a memory and one or more processors; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the device executes the method as described in any one of claims 1 to 15.

17. A system, characterized in that: include: a first device and a second device; The first device is used to perform the method as claimed in any one of claims 1 to 9; The second device is used to perform the method according to any one of claims 10-15.

18. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 15.

19. A computer program product, characterized in that The method comprises a computer program, which, when being executed on a computer, causes the computer to execute the method according to any one of claims 1 to 15.

Images