Information processing apparatus, information processing method, and information generation method

Abstract

A system for generating three-dimensional (3D) point cloud data includes at least one first processor configured to generate 3D point cloud data based on two-dimensional (2D) image data, where each point of the 3D point cloud data includes: position information including at least three coordinates indicative of a position of the point; object information marking the point as a first object selected from the plurality of objects; and category information that marks the point as belonging to a first category of a plurality of categories, where each category of the plurality of categories includes at least one respective object of the plurality of objects.

Description

[0001] Related Applications

[0002] This application claims the benefit of Japanese Priority Patent Application JP 2022-078633 filed on May 12, 2022, which is hereby incorporated by reference herein in its entirety. Technical Field

[0003] The present disclosure relates to an information processing device, an information processing method, and an information generating method. Background Art

[0004] Currently, regarding various systems such as autonomous driving, augmented reality (AR), and robots (e.g., SLAM: performing self-localization and map creation simultaneously), there is a device that processes an acquired three-dimensional point cloud (3D point cloud) in real time and performs object recognition (classification) for each point of the three-dimensional point cloud (e.g., see Patent Document 1). In such a technology, obstacles, background objects, etc. can be recognized in real time by performing classification using a neural network.

[0005] [Citation List]

[0006] [Patent Document]

[0007] [PT11]JP 2021-196829 A Summary of the invention

[0008] [Technical issues]

[0009] However, in the case of using the technology described above, three-dimensional point cloud data classified by predefined categories is required, and therefore the processing cost of generating a learning dataset is higher than that of a two-dimensional image dataset. In addition, in the case where the recognition labels of the category classification are different depending on the task, it is necessary to prepare a learning dataset and a learning model for each task, which increases the processing cost.

[0010] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information generating method that can reduce processing costs.

[0011] [Solution to the problem]

[0012] According to an embodiment of the present disclosure, an information processing device includes: a recognition unit, which performs recognition processing on an image and gives a recognition result for each image element of the image or each image element of interest; a classification unit, which classifies the recognition result of each image element of the image or each image element of interest according to a task, and gives a classification result for each image element of the image or each image element of interest; and a three-dimensional point cloud generation unit, which generates a three-dimensional point cloud associated with the image, and gives a classification result for each point of the three-dimensional point cloud based on the classification result of each image element of the image or each image element of interest.

[0013] According to an embodiment of the present disclosure, an information processing method includes: performing recognition processing on an image and giving a recognition result for each image element of the image or each image element of interest; classifying the recognition result of each image element of the image or each image element of interest according to a task, and giving a classification result for each image element of the image or each image element of interest; and generating a three-dimensional point cloud associated with the image, and giving a classification result for each point of the three-dimensional point cloud based on the classification result of each image element of the image or each image element of interest.

[0014] The information generating method according to an embodiment of the present disclosure includes generating information including position information of each point of a three-dimensional point cloud and a classification result of each point of the three-dimensional point cloud obtained by classifying a recognition result of each point of the three-dimensional point cloud according to a task.

[0015] A system for generating three-dimensional (3D) point cloud data, the system comprising: at least one first processor, which is configured to: generate 3D point cloud data based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: position information, the position information comprising at least three coordinates indicating the position of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories comprises at least one corresponding object among a plurality of objects.

[0016] A method for generating three-dimensional (3D) point cloud data, the method comprising: generating 3D point cloud data based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: position information, the position information comprising at least three coordinates indicating the position of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories comprises at least one corresponding object among a plurality of objects.

[0017] At least one non-transitory computer-readable storage medium having instructions encoded thereon, which, when executed by at least one processor, cause the at least one processor to perform a method for generating three-dimensional (3D) point cloud data, the method comprising: generating 3D point cloud data based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: position information, the position information comprising at least three coordinates indicating the position of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories comprises at least one corresponding object among the plurality of objects.

[0018] A system includes: at least one non-transitory computer-readable storage medium, the at least one non-transitory computer-readable storage medium having three-dimensional (3D) point cloud data encoded thereon, each point in the 3D point cloud data including: location information, the location information including at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories includes at least one corresponding object among a plurality of objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a diagram showing an example of a schematic configuration of an information processing system according to an embodiment of the present disclosure.

[0020] Figure 2 is a diagram showing an example of a schematic configuration of an information processing device according to an embodiment of the present disclosure.

[0021] Figure 3 is a flowchart showing an example of the flow of information processing according to an embodiment of the present disclosure.

[0022] Figure 4 is a diagram for describing an example of a flow of information processing according to an embodiment of the present disclosure.

[0023] Figure 5 1 is a diagram for describing a first specific example of identification label information and category label information according to an embodiment of the present disclosure.

[0024] Figure 6 2 is a diagram for describing a second specific example of identification label information and category label information according to an embodiment of the present disclosure.

[0025] Figure 7 2 is a diagram for describing a third specific example of identification label information and category label information according to an embodiment of the present disclosure.

[0026] Figure 8 : is a flowchart showing an example of the flow of a 3D point cloud generation process according to an embodiment of the present disclosure.

[0027] Fig. 9 3D point cloud generation processing according to an embodiment of the present disclosure is described in detail.

[0028] Fig.10 3D point cloud generation processing according to an embodiment of the present disclosure is described in detail.

[0029] Fig.11 is a diagram showing an example of a schematic configuration of a modification of the information processing apparatus according to the embodiment of the present disclosure.

[0030] Fig.12 is a diagram for describing a specific example of a graphical user interface (GUI) according to an embodiment of the present disclosure.

[0031] Fig.13 is a diagram showing an example of a schematic configuration of hardware according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the devices, systems, methods, etc. according to the present disclosure are not limited by the embodiments. In addition, in the following embodiments, the same components are basically represented by the same figure numerals, and redundant descriptions will be omitted.

[0033] One or more embodiments described below (including examples and variations) can be implemented independently of each other. On the other hand, at least a portion of the multiple embodiments described below can be appropriately combined with at least some other embodiments. Multiple embodiments can include novel features that are different from each other. Therefore, multiple embodiments can help solve different purposes or problems and can show different effects.

[0034] Furthermore, the present disclosure will be described according to the following item order.

[0035] 1. Implementation Method

[0036] 1-1. Configuration Example of Information Processing System

[0037] 1-2. Configuration Example of Information Processing Device

[0038] 1-3. Example of information processing

[0039] 1-4. Specific Examples of Identification Label Information and Category Label Information

[0040] 1-5. Example of 3D point cloud generation and processing

[0041] 1-6. Actions and Effects

[0042] 2. Another Configuration Example of Information Processing Device

[0043] 3. Specific embodiments of GUI

[0044] 4. Other Implementation Methods

[0045] 5. Hardware Configuration Example

[0046] 6. Appendix

[0047] <1. Implementation Method>

[0048] <1-1. Configuration Example of Information Processing System>

[0049] Reference Figure 1 A configuration example of the information processing system 1 according to the present embodiment is described. Figure 1 is a diagram showing an example of a schematic configuration of the information processing system 1 according to the present embodiment of the present disclosure.

[0050] like Figure 1 As shown, the information processing system 1 according to the present embodiment includes an information acquisition device 10 , an information processing device 20 , a server device 30 , and an application execution device (application execution device) 40 .

[0051] For example, the information acquisition device 10 is a device that acquires image information, distance information, etc. related to an image of a target object and transmits the acquired image information, distance information, etc. to the information processing device 20. The information acquisition device 10 is implemented, for example, by an imaging sensor such as an RGB sensor or an image sensor, or a distance measurement sensor such as a stereo camera or an indirect time-of-flight (iToF) sensor.

[0052] For example, an image sensor receives ambient light through a photodiode, detects RGB values, and acquires a color image. A stereo camera measures the distance to a target object (information in the depth direction of the target object) by photographing the target object from multiple different directions with a camera. An iToF sensor illuminates a target object with a periodic laser (continuous wave) and measures the distance to the target object based on the phase shift of the reflected light from the target object. However, the information acquisition device 10 may be a sensor other than an image sensor and a distance measurement sensor.

[0053] Here, in an imaging sensor, for example, a plurality of 2D images (two-dimensional images) are acquired. In this case, for example, two or more 2D image groups captured with overlapping areas are acquired as 2D images of a target object. In addition, in a distance measuring sensor, for example, a depth image (depth map) having depth information (distance information) of each image element is acquired. The depth image may be generated at least in part using a depth sensor. In some embodiments, a depth image is generated at least in part based on a first and a second set of 2D image data having corresponding fields of view that at least partially overlap.

[0054] The information processing device 20 performs recognition processing on the image based on the image information, distance information, etc. sent from the information acquisition device 10, gives a recognition result for each image element of the image (for example, a recognition tag indicating the recognition result, also referred to as object information in this article), classifies the recognition result of each image element of the image according to the task, and gives a classification result for each image element (for example, a category tag indicating the classification result, also referred to as category information in this article). In addition, the information processing device 20 generates a 3D point cloud with a classification result of each 3D point (three-dimensional point) based on the distance information, the classification result of each image element of the image, etc., and transmits point cloud information related to the 3D point cloud to the server device 30.

[0055] Here, for example, a 3D point cloud is a sample of the position and spatial structure of a target object (object). Typically, 3D point cloud data is acquired at a constant period of each frame time. By performing various types of computational processing on the 3D point cloud data, the exact position, posture, etc. of the target object can be detected (recognized).

[0056] The information processing device 20 is implemented, for example, by a processor such as a central processing unit (CPU) or a microprocessing unit (MPU). For example, the information processing device 20 is implemented by a processor that executes various programs using a random access memory (RAM) or the like as a work area. It should be noted that the information processing device 20 can be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Any CPU, MPU, ASIC and FPGA can be regarded as a controller. In addition, in addition to or instead of the CPU, the information processing device 20 can be implemented by a graphics processing unit (GPU). In addition, the information processing device 20 can be implemented by specific software rather than specific hardware.

[0057] The server device 30 stores and manages various types of information. For example, the server device 30 stores and manages point cloud information transmitted from the information processing device 20. The server device 30 is implemented by a server such as a cloud server, a PC server, a medium-sized server, a mainframe server, etc.

[0058] Here, the sending and receiving of data between the server device 30 is performed, for example, via a network. The network is, for example, a communication network (communication link), such as a local area network (LAN), a wide area network (WAN), a cellular network, a fixed-line link, a regional Internet Protocol (IP) link, or the Internet. The network may include a wired network or a wireless network. In addition, the network may include a core network. The core network is, for example, an evolved packet core (EPC) or a 5G core network (5GC). In addition, the network may include a data network other than the core network. For example, the data network may be a service network of a telecommunications operator, for example, an IP multimedia subsystem (IMS) network. In addition, the data network may also be a private network, such as an internal company network.

[0059] It should be noted that as a radio access technology (RAT), Long Term Evolution (LTE), New Radio (NR), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. can be used. Several types of radio access technologies can be used, for example, NR and Wi-Fi can be used, and LTE and NR can be used. LTE and NR are cellular communication technologies that realize mobile communication by arranging multiple areas covered by base stations in a honeycomb shape.

[0060] The application execution device 40 executes the application using the point cloud information transmitted from the server device 30. As the application, for example, various applications such as a general-purpose application and a dedicated application can be used.

[0061] Similar to the information processing device 20, the application execution device 40 is implemented by a processor such as a CPU or an MPU, for example. For example, the application execution device 40 is implemented by a processor that executes various programs using a RAM or the like as a work area. It should be noted that the application execution device 40 may be implemented by an integrated circuit such as an ASIC or an FPGA. Furthermore, in addition to or instead of a CPU, the application execution device 40 may be implemented by a GPU. In addition, the application execution device 40 may be implemented by specific software rather than specific hardware.

[0062] Here, the application execution device 40 is installed on various devices such as cars, robots, and user terminals. However, in addition to the application execution device 40, one or both of the information acquisition device 10 and the information processing device 20 may be appropriately installed on various devices. The user terminal is a terminal used by a user, and receives point cloud information from the server device 30, for example. The user terminal is implemented, for example, by a terminal such as a personal computer (e.g., a notebook computer or a desktop computer), a smart device (e.g., a smart phone or a tablet), or a personal digital assistant (PDA). In addition, the user terminal can be implemented by, for example, an xR device (such as an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device). The xR device may be a glasses-type device (e.g., AR / MR / VR glasses) or a head-mounted or goggle-type device (e.g., AR / MR / VR headsets, AR / MR / VR goggles). These xR devices can display video for only one eye or can display video for both eyes.

[0063] It should be noted that the application execution device 40 receives the point cloud information from the information processing device 20 via the server device 30, but is not limited to this, and for example, the point cloud information may be received directly from the information processing device 20 without passing through the server device 30. This is appropriately selected according to the configuration, purpose, etc. of the various devices on which the application execution device 40 is installed.

[0064] <1-2. Configuration Example of Information Processing Device>

[0065] Reference Figure 2 A configuration example of the information processing device 20 according to the present embodiment is described. Figure 2 : is a diagram showing an example of a schematic configuration of the information processing device 20 according to the present embodiment.

[0066] like Figure 2 As shown, the information processing device 20 includes a recognition unit 21 , a learning model 22 , a classification unit 23 , and a 3D point cloud generation unit (three-dimensional point cloud generation unit) 24 .

[0067] The recognition unit 21 performs recognition processing on the image and gives a recognition result for each image element of the image. For example, the recognition unit 21 performs recognition processing for each image element to recognize the attributes of the image element and gives a recognition result (e.g., a recognition label). The recognition result is the result of recognizing what the target object is, and indicates, for example, a car, a person, a building, etc. For example, the recognition unit 21 provides an recognition label for each image element based on the learning model 22. It should be noted that the recognition unit 21 can also provide an recognition label for each image element of interest rather than for each image element. The image element of interest is not all the image elements of the image, but, for example, a specific image element set in advance.

[0068] The learning model 22 is an identification label learning model that determines any identification label as an identification result from a plurality of types of identification labels. The learning model 22 is, for example, a model that uses data given in advance by identification labels (correct answer labels) as teacher data, learns a neural network (e.g., CNN: convolutional neural network, etc.), which is one of machine learning that recognizes a target object from an image, and performs recognition processing on the image to provide an identification label for each image element of the image.

[0069] The classification unit 23 classifies the recognition result of each image element of the image according to the task, and gives the classification result (for example, a category label) for each image element of the image. The classification result is a result of identifying which category the recognition result is classified into, and indicates, for example, category 1, category 2, category 3, etc. The classification can be performed using a learning model such as a machine learning model. It should be noted that as tasks, various tasks such as a task of wanting to know a soil and sand area, a task of wanting to know a vehicle area, etc. can be listed.

[0070] The 3D point cloud generation unit 24 generates a 3D point cloud (also referred to herein as 3D point cloud data) having a classification result for each 3D point based on the distance information associated with the image and the classification result for each image element of the image. The 3D point cloud is a collection of points having position information (e.g., three-dimensional coordinates X, Y, Z) and having a classification result for each position.

[0071] For example, the 3D point cloud generation unit 24 generates a 3D point cloud based on the distance information of each image element of the 2D image group (e.g., multiple sets of 2D image data, such as the first and second sets of 2D image data) or the depth image. It should be noted that as a method of generating a 3D point cloud from a 2D image group, for example, stereo matching or structure from motion (SfM) and multi-view stereo (MVS) are used. SfM generates low-density point cloud data recovered from feature points by triangulation. MVS generates high-density point cloud data.

[0072] It should be noted that each block constituting the above-mentioned information processing device 20 (for example, the recognition unit 21, the learning model 22, the classification unit 23, and the 3D point cloud generation unit 24) is a functional block indicating the function of the information processing device 20. These functional blocks can be software blocks or hardware blocks. For example, each block can be a software module implemented by software (microprogram) or a circuit block on a semiconductor chip (bare die). Of course, each block can be a processor or an integrated circuit. In addition, the information processing device 20 may include functional units different from the above-mentioned blocks. The configuration method of each block is arbitrary. In addition, part or all of the operations of each block can be performed by another device.

[0073] <1-3. Example of information processing>

[0074] Reference Figure 3 and Figure 4 An example of information processing according to the present embodiment is described. Figure 3 : is a flowchart showing an example of the flow of information processing according to the present embodiment. Figure 4 is a diagram for describing an example of the flow of information processing according to the present embodiment.

[0075] like Figure 3 As shown, in step S11, the information acquisition device 10 continuously performs shooting in a manner where there are overlapping areas, and acquires multiple 2D images (e.g., color images). In step S12, the recognition unit 21 performs object recognition processing on each 2D image, and gives a recognition result (e.g., recognition label) for each image element of each 2D image. In step S13, the classification unit 23 classifies the recognition result of each image element of each 2D image, and gives a classification result (e.g., category label) for each image element of each 2D image. In step S14, the 3D point cloud generation unit 24 generates a 3D point cloud with the classification result of each 3D point (e.g., category 1, category 2, etc.).

[0076] like Figure 4 As shown in , in step S11, a group of images captured with an overlapping area (overlap), that is, two or more 2D images, are acquired. In this case, for example, the information acquisition device 10 is installed on a mobile body such as a drone, an airplane, or a helicopter equipped with an aerial camera. For example, the transmission of image information can be achieved through a radio access technology such as Wi-Fi (registered trademark).

[0077] Next, in step S12, using the learning model 22, recognition processing is performed on each 2D image, and a recognition label is provided for each image element of each 2D image. Therefore, in each 2D image, a recognition label map (also referred to as a category label map in this article) having a recognition label for each image element is generated. The recognition label map is an N-value image as a result of 2D segmentation. Figure 4 In the embodiment of , the identification tags include "soil", "vegetation", "asphalt", "water", "construction equipment", "building", "object", "person" and "car". In this case, because there are 9 identification tags, the identification tag map is indicated by 9 colors and becomes a 9-value image (N=9).

[0078] Next, in step S13, the identification labels of each image element of each 2D image are classified according to the task, and a category label (classification label) is provided for each image element of these 2D images. Thus, a category label map having a category label of each image element is generated. The category label map is an M-value image, which is a 2D segmentation result (where M < N). Figure 4 In the embodiment of the present invention, as the category labels corresponding to the tasks, there are survey targets (category 1) and non-survey targets (category 2). The survey targets are "soil", and the non-survey targets are "vegetation", "asphalt", "water", "construction equipment", "building", "article", "person", and "car". In this case, since the identification labels of each image element of each 2D image are classified into two categories, the category label map is indicated by two colors (e.g., white and black) and becomes a binary image (M=2).

[0079] Finally, in step S14, a 3D point cloud having a category label for each point of the 3D point cloud is generated, that is, a 3D point cloud with category labels. For example, a 3D point cloud is generated, a category label is provided for each point of the 3D point cloud, and a 3D point cloud with category labels is generated. In the process of providing the category label, the category label of each image element of the 2D image is provided to the 3D point cloud corresponding to each image element. This 3D point cloud generation process will be described in detail later. It should be noted that in Figure 4 In the embodiment, there are various fill areas such as black solid fill areas and dot fill areas, and these different fill areas indicate different types of category labels (eg, category 1, category 2, etc.).

[0080] Here, for example, in the case where the identification label of a certain image element is "soil" in step S13, the identification label is classified as a survey target (category 1), and "category 1" is provided as a category label to the image element. In addition, in the case where the identification label of another image element is "vegetation", the identification label is classified as a non-survey target (category 2), and "category 2" is provided as a category label to the image element. Such processing is performed on all image elements, and a category label is provided for each image element. Thereafter, in step S14, "category 1" is provided as a category label to the 3D point corresponding to the image element providing "category 1". In addition, "category 2" is provided as a category label to the 3D point corresponding to the image element providing "category 2". Such processing is performed on all image elements, and a category label is provided for each 3D point cloud. It should be noted that the 3D point corresponding to a specific image element is, for example, a 3D point at the position of the 3D coordinates calculated by calculating the 3D coordinates of the image element.

[0081] According to such a series of information processing, recognition processing is performed on multiple 2D images that are the source of 3D point cloud generation, and then classification processing is performed to classify the recognition labels of each image element of each 2D image according to the task, and a category label is provided for each image element of each 2D image. In the recognition processing, a learning model 22 including all assumed recognition labels is prepared, and recognition processing is performed based on the learning model 22. Generally, a learning model is prepared for each task, and when the task changes, the recognition processing of the learning model corresponding to the task is performed. However, according to the present embodiment, when the task changes, the classification processing corresponding to the task is performed. That is, even if the task is changed, the classification processing according to the task is performed in a manner that the point cloud information can be generated according to various tasks. It should be noted that in the classification processing, a correspondence indicating which category is required for each task, that is, the category label information of each task is prepared.

[0082] <1-4. Specific Examples of Identification Tag Information and Category Tag Information>

[0083] Will see Figures 5 to 7 Specific examples of identification label information and category label information according to the present embodiment are described. Figures 5 to 7 is a diagram for describing a specific embodiment of identification label information and category label information.

[0084] In recognition processing and classification processing (see Figure 4 In steps S12 and S13 in the above method, a 2D image is used as input data, a recognition label map (N value image) is intermediately output, and a category label map (M value image, where M < N) is output. In the first and second specific embodiments, the input 2D image is a spatial image, and in the third specific embodiment, the input 2D image is a vehicle-mounted image. It should be noted that the 2D image can be an RGB color image or a grayscale image.

[0085] like Figure 5 As shown in the first specific embodiment (input spatial image: N = 9, M = 2), there are nine types of identification labels, namely "soil", "vegetation", "asphalt", "water", "construction equipment", "building", "object", "person" and "car". There are two types of category labels: survey target (category 1) and non-survey target (category 2). Figure 5 In the embodiment of the present invention, the survey target is "soil", and the non-survey targets are "vegetation", "asphalt", "water", "construction equipment", "building", "object", "person" and "car".

[0086] like Figure 6 As shown in the second specific embodiment (input spatial image: N = 9, M = 3 embodiment), Figure 5Similarly, there are nine types of identification tags, "soil", "vegetation", "asphalt", "water", "construction equipment", "building", "object", "person", and "car". There are three types of category tags, terrain (category 1), moving objects (category 2), and non-moving objects (category 3). Figure 6 In the embodiment of FIG. 5 , the terrain is “soil”, “vegetation”, and “water”, the moving targets are “construction equipment”, “cars”, and “people”, and the non-moving targets are “objects”, “buildings”, and “asphalt”.

[0087] like Figure 7 As shown, in the third specific embodiment (the embodiment with input vehicle images: N=7, M=3), there are seven types of identification labels, "tram", "person", "car", "telephone pole", "signal", "road" and "sky". There are three types of category labels, contact attention target (category 1), traffic rule recognition target (category 2) and non-recognition target (category 3). Figure 7 In the embodiment, the contact attention targets are "tram", "person", "car" and "telephone pole", the traffic rule recognition targets are "signal" and "road", and the non-recognition target is "sky".

[0088] According to the first specific embodiment, the second specific embodiment, and the third specific embodiment, category label information indicating the relationship between the recognition label and the category label may be set for a specific task, or may be set for each task. In the case where category label information is set for each task, the category label information is selected according to the task and used for classification processing. By preparing category label information for each task in this way, various tasks can be processed. Generally, it is necessary to prepare a learning model 22 for each task, but according to the present embodiment, if one learning model 22 is prepared, the recognition result based on the learning model 22 is converted into a classification result according to the task in a manner that can cope with various tasks.

[0089] It should be noted that according to Figures 5 to 7 The specific embodiment of is only an embodiment, and the category label information corresponding to other tasks can also be used. For example, the classification unit 23 can store the category label information for each task in a storage unit. The storage unit can be set in the classification unit 23, or can be set outside the classification unit 23. The storage unit is implemented by, for example, a storage device capable of reading and writing data, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, or a hard disk.

[0090] <1-5. Example of 3D point cloud generation processing>

[0091] Will see Figures 8 to 10An example of 3D point cloud generation processing according to the present embodiment will be described.

[0092] Figure 8 : is a flowchart showing an example of the flow of 3D point cloud generation processing according to the present embodiment.

[0093] Fig. 9 and Fig.10 3D point cloud generation processing according to the present embodiment is described in detail.

[0094] like Figure 8 As shown, in step S21, the 3D point cloud generation unit 24 calculates the 3D coordinates corresponding to the pixel (image element). In step S22, the 3D point cloud generation unit 24 gives a classification result (e.g., a category label) corresponding to the point (three-dimensional point) at the 3D position based on the 3D coordinates. In step S23, the 3D point cloud generation unit 24 determines whether all pixels have been scanned, and when it is determined that all pixels have not been scanned (No in step S23), the process is returned to step S21, and when it is determined that all pixels have been scanned (Yes in step S23), the process is terminated.

[0095] In this treatment, Fig. 9 As shown in , two or more class label maps (M-value images), depth images corresponding to these maps, and shooting position information corresponding to each image are input. The depth image is an image with a depth value (e.g., d1, d2, etc.) as the distance information of each image element. The shooting position information is the position of the camera used for shooting, and is information indicating that the camera has been calibrated, for example. A 3D point cloud with class labels is generated based on these pieces of information, and point cloud information related to the 3D point cloud is output.

[0096] Here, each point of the 3D point cloud configures point cloud information in a data format that includes at least (X, Y, Z, C). XYZ is the 3D coordinate of the 3D point, and C is the category label value (e.g., category 1, category 2, category 3, etc.). That is, the point cloud information includes at least the position information (X, Y, Z) and classification result information (C) of each point of the 3D point cloud.

[0097] In addition, for example, the point cloud information may be in a format that reflects the color information of the original image. That is, for each point of the 3D point cloud, the point cloud information may be configured in a data format including at least (X, Y, Z, R, G, B, C). RGB is the color information of the point (e.g., 256 values). That is, the point cloud information may include, for example, position information (X, Y, Z), color information (R, G, B), and classification result information (C) for each point of the 3D point cloud.

[0098] It should be noted that with respect to the above camera calibration, Fig.10As shown in FIG. 1 , the relationship between the coordinate points of the image and the world coordinate points (3D positions) is described by the internal parameter matrix and the external parameter matrix. For example, the 3D position (world coordinate point) as the camera position is determined by a relational expression. Fig.10 In an embodiment of the present invention, the internal parameters are the transformation from the camera coordinate system to the image coordinate system. The external parameters are the transformation from the world coordinate system to the camera coordinate system. The internal parameters and the external parameters are, for example, parameters of a perspective projection model.

[0099] <1-6. Actions and Effects>

[0100] As described above, according to the present embodiment, the information processing device 20 includes: a recognition unit 21 that performs recognition processing on an image and gives a recognition result (e.g., a recognition label) for each image element of the image or each image element of interest; a classification unit 23 that classifies the recognition result (e.g., a category label) of each image element of the image or each image element of interest according to a task and gives a classification result for each image element of the image or each image element of interest; and a 3D point cloud generation unit 24 that generates a 3D point cloud related to the image and provides a classification result for each point of the 3D point cloud according to the classification result of each image element of the image or each image element of interest. Therefore, after the recognition processing, a classification processing of classifying the recognition result of each image element of the image or each image element of interest is performed according to the task, and a classification result is provided for each image element of the image or each image element of interest. Therefore, for example, in the case of changing the task, it is only necessary to perform the classification processing according to the task without re-executing the recognition processing that is conventionally required, and the processing cost can be compressed.

[0101] In addition, the recognition unit 21 may perform recognition processing on a plurality of 2D images as images, and may give a recognition result of each image element of each 2D image or each image element of interest, the classification unit 23 may classify the recognition result of each image element of each 2D image or each image element of interest, and may give a classification result of each image element of each 2D image or each image element of interest, and the 3D point cloud generation unit 24 may generate a 3D point cloud based on distance information of each image element of each 2D image or each target image element, and may give a classification result of each point of the 3D point cloud based on the classification result of each image element or each target image element in the plurality of 2D images. Therefore, a 3D point cloud having a classification result of each 3D point can be reliably generated.

[0102] In addition, the recognition unit 21 may perform recognition processing on the depth image as an image, and may give a recognition result of each image element of the depth image or each image element of interest, the classification unit 23 may classify the recognition result of each image element of the depth image or each image element of interest, and may give a classification result of each image element of the depth image or each image element of interest, and the 3D point cloud generation unit 24 may generate a 3D point cloud based on the distance information of each image element of the depth image or each target image element, and may give a classification result of each point of the 3D point cloud based on the classification result of each image element of the depth image or each target image element. Therefore, a 3D point cloud having a classification result of each 3D point can be reliably generated.

[0103] In addition, the recognition unit 21 can provide a determined recognition label for each image element or each image element of interest in the image based on the learning model 22, and the learning model 22 determines one recognition label as the recognition result from a plurality of types of recognition labels indicating the recognition result. Therefore, it is possible to reliably provide a recognition label for each image element or each image element of interest in the image.

[0104] In addition, the recognition unit 21 may give an identification tag indicating the recognition result as the recognition result, the classification unit 23 may give a category tag indicating the classification result as the classification result, and the types of category tags may be less than the types of recognition tags. For example, there may be fewer possible category tags than possible recognition tags. Therefore, the processing cost can be reliably compressed.

[0105] In addition, the classification unit 23 may generate a class label map having a class label for each image element of the image or each image element of interest, and the 3D point cloud generation unit 24 may provide a class label for each point of the 3D point cloud based on the class label map. Therefore, a 3D point cloud having a class label for each 3D point can be reliably generated.

[0106] In addition, the classification unit 23 may select a category label information from the category label information of each task indicating the relationship between the recognition label and the category label, and may provide a category label for each image element of the image or each image element of interest based on the selected category label information. Therefore, a 3D point cloud having a category label for each 3D point can be reliably generated.

[0107] <2. Another Configuration Example of Information Processing Device>

[0108] Reference Fig.11 Another configuration example of the information processing device 20 according to the above-described embodiment is described. Fig.11 : is a diagram showing an example of a schematic configuration of a modification of the information processing apparatus 20 according to the present embodiment.

[0109] like Fig.11 As shown, the information processing device 20 may include a recognition unit 21, a learning model 22, and a classification unit 23, and the server device 30 may include a 3D point cloud generation unit 24. According to this configuration, heavy 3D point cloud generation processing can be performed on the server device 30 (e.g., a cloud server) with abundant computing resources. It should be noted that at least the class label map (M value image) is sent from the information processing device 20 to the server device 30. For example, multiple class label maps are sent.

[0110] <3. Specific Implementation of GUI>

[0111] Reference Fig.12 A specific example of a GUI (Graphical User Interface) according to the embodiment is described. Fig.12 is a diagram for describing a specific example of a GUI according to an embodiment.

[0112] like Fig.12 As shown in FIG. 1 , the user terminal 100 includes a display unit 110. The display unit 110 is, for example, a liquid crystal display or an organic electroluminescence (EL) display, and a touch panel is used as the display unit 110.

[0113] The display unit 110 includes a display area 111, a category selection GUI 112, and a task selection GUI 113. The display unit 110 displays a 3D point cloud in the display area 111 based on the point cloud information. The category selection GUI 112 includes a plurality of check boxes for selecting target categories. The user checks the check boxes and selects one or more target categories. Fig.12 In the embodiment of , there are categories 1 to 5, etc. as categories. It should be noted that category selection corresponds to setting a task including the selected category. The task selection GUI 113 includes a plurality of check boxes for selecting target tasks instead of categories. The user checks the check boxes and selects one or more target tasks. When a task is selected, a target category corresponding to the task is automatically set. Fig.12 In the embodiment, Task 1 to Task 3 are tasks.

[0114] The user operates the category selection GUI 112 to select a target category, or operates the task selection GUI 113 to select a target task. In some embodiments, the selection of the target object can be performed. In response to this, for example, the user terminal 100 sends information such as a target category, a target object, or a task to the information processing device 20 via the server device 30. The 3D point cloud generation unit 24 receives the sent information such as the category, object, and / or task of the target, and sends a 3D point cloud corresponding to the category, object, and / or task of the target to the user terminal 100 via the server device 30, that is, point cloud information related to the 3D point cloud with the classification result (e.g., category label) and / or recognition result (e.g., recognition label) of each point. Afterwards, the display unit 110 displays the 3D point cloud in the display area 111 based on the transmitted point cloud information. In this way, the 3D point cloud generated according to the selection of the target is displayed.

[0115] <4. Other Implementation Methods>

[0116] The processing according to the above embodiment (or variant) can be performed in various different modes (modifications) different from the above embodiment. For example, in the processing described in the above embodiment, all or part of the processing described as being automatically performed can be performed manually, or all or part of the processing described as being manually performed can be performed automatically by known methods. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown in the specification and drawings can be arbitrarily changed. For example, the various types of information shown in the figures are not limited to the information shown.

[0117] In addition, each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific form of distribution and integration of each device is not limited to the form shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0118] In addition, the above-mentioned embodiments (or modified examples) can be appropriately combined within the range that does not contradict the processing contents. In addition, the effects described in this specification are only examples and are not limited, and other effects can be provided.

[0119] <5. Hardware Configuration Example>

[0120] A specific hardware configuration example of information devices such as the information processing device 20, the server device 30, and the application execution device 40 according to the above-mentioned embodiment (or modification) will be described. The information devices such as the information processing device 20, the server device 30, the application execution device 40 of the embodiment (or modification) may be, for example, composed of a computer having Fig.13The configuration shown is implemented by a computer 500 . Fig.13 is a diagram showing an example of a schematic configuration of hardware that realizes the functions of an information device.

[0121] like Fig.13 As shown, the computer 500 includes a CPU 510, a RAM 520, a read only memory (ROM) 530, a hard disk drive (HDD) 540, a communication interface 550, and an input / output interface 560. Each unit of the computer 500 is connected through a bus 570.

[0122] The CPU 510 operates based on the program stored in the ROM 530 or the HDD 540, and controls each unit. For example, the CPU 510 develops the program stored in the ROM 530 or the HDD 540 in the RAM 520, and executes processing corresponding to the various programs.

[0123] The ROM 530 stores a boot program such as a basic input output system (BIOS) executed by the CPU 510 when the computer 500 is activated, a program depending on the hardware of the computer 500 , and the like.

[0124] The HDD 540 is a recording medium that can be read by the computer 500 and performs non-transitory recording of programs executed by the CPU 510, data used by such programs, etc. Specifically, the HDD 540 is a recording medium that records the information processing program according to the present disclosure as an embodiment of the program data 541.

[0125] The communication interface 550 is an interface for connecting the computer 500 to an external network 580 (eg, the Internet). For example, the CPU 510 receives data from another device via the communication interface 550 or transmits data generated by the CPU 510 to another device.

[0126] The input / output interface 560 is an interface for connecting the input / output device 590 and the computer 500. For example, the CPU 510 receives data from an input device such as a keyboard or a mouse via the input / output interface 560. In addition, the CPU 510 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 560.

[0127] It should be noted that, in addition, the input / output interface 560 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. For example, the medium is an optical recording medium such as a digital versatile disk (DVD) or a phase-change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0128] Here, for example, in the case where the computer 500 is used as an information device such as the information processing device 20, the server device 30, or the application execution device 40 according to each embodiment (or modification), the CPU 510 of the computer 500 executes the information processing program loaded on the RAM 520 to implement all or part of the functions of the corresponding units such as the information processing device 20, the server device 30, or the application execution device 40 according to each embodiment (or modification). In addition, the HDD 540 stores the information processing program and data according to each embodiment. It should be noted that the CPU 510 reads the program data 541 from the HDD 540 and executes the program data, but as another embodiment, these programs can be obtained from another device via the external network 580.

[0129] <6. Appendix>

[0130] It should be noted that the present technology can also have the following configurations.

[0131] (1) An information processing device, comprising: a recognition unit, which performs recognition processing on an image and gives a recognition result for each image element of the image or each image element of interest; a classification unit, which classifies the recognition result of each image element of the image or each image element of interest according to a task and gives a classification result for each image element of the image or each image element of interest; and a three-dimensional point cloud generation unit, which generates a three-dimensional point cloud associated with the image and gives a classification result for each point of the three-dimensional point cloud based on the classification result of each image element of the image or each image element of interest.

[0132] (2) An information processing device according to (1), wherein a recognition unit performs recognition processing on a plurality of two-dimensional images as images and gives a recognition result for each image element in the plurality of two-dimensional images or each image element of interest, a classification unit classifies the recognition result for each image element in the plurality of two-dimensional images or each image element of interest and gives a classification result for each image element in the plurality of two-dimensional images or each image element of interest, and a three-dimensional point cloud generation unit generates a three-dimensional point cloud based on distance information of each image element in the plurality of two-dimensional images or each image element of interest, and gives a classification result for each point of the three-dimensional point cloud based on the classification result for each image element in the plurality of two-dimensional images or each image element of interest.

[0133] (3) An information processing device according to (1), wherein a recognition unit performs recognition processing on a depth image as an image and gives a recognition result for each image element of the depth image or each image element of interest, a classification unit classifies the recognition result of each image element of the depth image or each image element of interest and gives a classification result for each image element of the depth image or each image element of interest, and a three-dimensional point cloud generation unit generates a three-dimensional point cloud based on distance information of each image element of the depth image or each target image element, and gives a classification result for each point of the three-dimensional point cloud based on the classification result of each image element of the depth image or each target image element.

[0134] (4) An information processing device according to any one of (1) to (3), wherein the recognition unit provides a determined recognition label for each image element of the image or each image element of interest based on a learning model, and the learning model determines an recognition label from multiple types of recognition labels indicating the recognition results as the recognition result.

[0135] (5) An information processing device according to any one of (1) to (4), wherein the recognition unit gives an recognition label indicating the recognition result as the recognition result, and the classification unit gives a category label indicating the classification result as the classification result, and the types of category labels are less than the types of recognition labels.

[0136] (6) An information processing device according to (5), wherein the classification unit generates a category label map with a category label for each image element of the image or each image element of interest, and the three-dimensional point cloud generation unit provides a category label for each point of the three-dimensional point cloud based on the category label map.

[0137] (7) An information processing device according to (5) or (6), wherein the classification unit selects a category label information from the category label information of each task for indicating the relationship between the identification label and the category label, and provides a category label for each image element of the image or each image element of interest based on the selected category label information.

[0138] (8) An information processing method, comprising: performing recognition processing on an image and giving a recognition result for each image element of the image or each image element of interest; classifying the recognition result for each image element of the image or each image element of interest according to a task, and giving a classification result for each image element of the image or each image element of interest; and generating a three-dimensional point cloud associated with the image, and giving a classification result for each point of the three-dimensional point cloud based on the classification result for each image element of the image or each image element of interest.

[0139] (9) An information generating method comprising: generating information including position information of each point of a three-dimensional point cloud and a classification result of each point of the three-dimensional point cloud obtained by classifying a recognition result of each point of the three-dimensional point cloud according to a task.

[0140] (10) An information processing system including the information processing device according to any one of (1) to (7).

[0141] (11) An information processing method using the information processing apparatus according to any one of (1) to (7).

[0142] (12) An information generating method using the information processing device according to any one of (1) to (7).

[0143] (13) A system for generating three-dimensional (3D) point cloud data, the system comprising: at least one first processor, configured to: generate 3D point cloud data based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: position information, the position information comprising at least three coordinates indicating the position of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories comprises at least one corresponding object among the plurality of objects.

[0144] (14) A system according to (13), wherein the plurality of categories are defined based on tasks.

[0145] (15) The system of any one of (13) to (14), wherein the number of the plurality of categories is less than the number of the plurality of objects.

[0146] (16) The system according to any one of (13) to (15), further including a memory storing a table including a label of each of a plurality of categories and a label of each of a plurality of objects.

[0147] (17) A system according to any one of (13) to (16), wherein at least one first processor is further configured to: receive 2D image data; perform 2D object recognition processing on the 2D image data to generate labeled 2D image data including object information; classify the labeled 2D image data to generate classified 2D image data including category information; and convert the classified 2D image data to generate 3D point cloud data including position information, object information, and category information.

[0148] (18) A system according to (17), wherein at least one first processor is configured to perform 2D object recognition processing at least in part using a machine learning model.

[0149] (19) The system of (17), wherein the at least one first processor is configured to classify the labeled 2D image data using a machine learning model.

[0150] (20) A system according to any one of (13) to (19), wherein the 2D image data includes multiple 2D image data, the multiple 2D image data includes a first 2D image data set and a second 2D image data set, wherein a field of view of the first 2D image data set at least partially overlaps a field of view of the second 2D image data set, and at least one processor is configured to generate 3D point cloud data based on the multiple 2D image data.

[0151] (21) The system of (20), wherein the at least one first processor is further configured to receive depth image data, the field of view of the depth image data at least partially overlapping the field of view of the 2D image data.

[0152] (22) A system according to (21), wherein the depth image data is generated by a depth sensor.

[0153] (23) A system according to (21), wherein the depth image data is generated based on a plurality of 2D image data.

[0154] (24) A system according to any one of (13) to (23), wherein the 2D image data is generated by a camera.

[0155] (25) A system according to any one of (13) to (24), wherein the 2D image data includes a plurality of pixels, and the at least one first processor is further configured to generate a category label map having category information for each pixel of the 2D image data, and wherein the at least one first processor is configured to generate 3D point cloud data based on the category map.

[0156] (26) The system of any one of (13) to (25), wherein the at least one first processor is further configured to display the 3D point cloud data on a display.

[0157] (27) A system according to any one of (13) to (26), wherein at least one first processor is configured to display 3D point cloud data based on a selection of one or more objects among a plurality of objects and / or a selection of one or more categories among a plurality of categories.

[0158] (28) A method for generating three-dimensional (3D) point cloud data, the method comprising: generating 3D point cloud data based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: position information, the position information comprising at least three coordinates indicating the position of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories comprises at least one corresponding object among the plurality of objects.

[0159] (29) According to the method of (28), it further includes: receiving 2D image data; performing 2D object recognition processing on the 2D image data to generate labeled 2D image data including object information; classifying the labeled 2D image data to generate classified 2D image data including category information; and converting the classified 2D image data to generate 3D point cloud data including position information, object information and category information.

[0160] (30) A method according to (29), wherein the 2D object recognition processing is performed at least in part using a machine learning model.

[0161] (31) According to the method of (29) or (30), classifying the labeled 2D image data includes classifying the labeled 2D image data using a machine learning model.

[0162] (32) at least one non-transitory computer-readable storage medium having instructions encoded thereon, the instructions, when executed by at least one processor, causing the at least one processor to perform a method for generating three-dimensional (3D) point cloud data, the method comprising: generating 3D point cloud data based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: location information, the location information comprising at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories comprises at least one corresponding object among the plurality of objects.

[0163] (33) A system comprising: at least one non-transitory computer-readable storage medium having three-dimensional (3D) point cloud data encoded thereon, each point in the 3D point cloud data comprising: location information, the location information comprising at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and category information, marking the point as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories includes at least one corresponding object among a plurality of objects.

[0164] [Reference Numbers List]

[0165] 1 Information Processing System

[0166] 10 Information acquisition equipment

[0167] 20 Information processing equipment

[0168] 21 Identification Unit

[0169] 22 Learning Model

[0170] 23 Taxa

[0171] 24 3D point cloud generation unit

[0172] 30 Server Equipment

[0173] 40 Application execution device

[0174] 100 User Terminals

[0175] 110 Display unit

[0176] 111 Display Area

[0177] 112 Category Selection GUI

[0178] 113 Task Selection GUI

[0179] 500 computers.

Claims

1. A system for generating three-dimensional (3D) point cloud data, the system comprising: At least one first processor is configured to: The 3D point cloud data is generated based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: location information, the location information comprising at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and The point is marked as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories includes at least one corresponding object among the plurality of objects.

2. The system according to claim 1, wherein: The plurality of categories are defined based on tasks.

3. The system according to claim 1, wherein: The number of the plurality of categories is less than the number of the plurality of objects. 4 . The system of claim 1 , further comprising a memory storing a table including a label for each of the plurality of categories and a label for each of the plurality of objects.

5. The system according to claim 1, wherein: The at least one first processor is further configured to: receiving the 2D image data; performing 2D object recognition processing on the 2D image data to generate marked 2D image data including the object information; classifying the labeled 2D image data to generate classified 2D image data including the category information; as well as The classified 2D image data is converted to generate the 3D point cloud data including the position information, the object information, and the category information.

6. The system according to claim 5, wherein: The at least one first processor is configured to perform the 2D object recognition processing at least in part using a machine learning model.

7. The system according to claim 5, wherein: The at least one first processor is configured to classify the labeled 2D image data using a machine learning model.

8. The system according to claim 1, wherein: The 2D image data includes a plurality of 2D image data, the plurality of 2D image data includes a first 2D image data set and a second 2D image data set, wherein a field of view of the first 2D image data set at least partially overlaps a field of view of the second 2D image data set, and at least one processor is configured to generate the 3D point cloud data based on the plurality of 2D image data.

9. The system according to claim 8, wherein: The at least one first processor is further configured to receive depth image data, the field of view of the depth image data at least partially overlapping the field of view of the 2D image data.

10. The system according to claim 9, wherein: The depth image data is generated by a depth sensor.

11. The system according to claim 9, wherein: The depth image data is generated based on the plurality of 2D image data.

12. The system of claim 1, wherein: The 2D image data is generated by a camera.

13. The system of claim 1, wherein: The 2D image data includes a plurality of pixels, and the at least one first processor is further configured to generate a category label map having the category information for each pixel of the 2D image data, and wherein the at least one first processor is configured to generate the 3D point cloud data based on the category map.

14. The system of claim 1, wherein: The at least one first processor is further configured to display the 3D point cloud data on a display.

15. The system of claim 1, wherein: The at least one first processor is configured to display the 3D point cloud data based on a selection of one or more objects of the plurality of objects and / or a selection of one or more categories of the plurality of categories.

16. A method for generating three-dimensional (3D) point cloud data, the method comprising: The 3D point cloud data is generated based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: location information, the location information comprising at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and The point is marked as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories includes at least one corresponding object among the plurality of objects.

17. The method according to claim 16, further comprising: receiving the 2D image data; performing 2D object recognition processing on the 2D image data to generate marked 2D image data including the object information; classifying the labeled 2D image data to generate classified 2D image data including the category information; as well as The classified 2D image data is converted to generate the 3D point cloud data including the position information, the object information, and the category information.

18. The method according to claim 17, wherein: The 2D object recognition process is performed at least in part using a machine learning model.

19. The method according to claim 17, wherein: Classifying the labeled 2D image data includes using a machine learning model to classify the labeled 2D image data.

20. At least one non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by at least one processor, cause the at least one processor to perform a method for generating three-dimensional (3D) point cloud data, the method comprising: The 3D point cloud data is generated based on two-dimensional (2D) image data, wherein each point in the 3D point cloud data comprises: location information, the location information comprising at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and The point is marked as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories includes at least one corresponding object among the plurality of objects.

21. A system comprising: At least one non-transitory computer-readable storage medium having encoded thereon three-dimensional (3D) point cloud data, each point in the 3D point cloud data comprising: location information, the location information comprising at least three coordinates indicating the location of the point; object information, marking the point as a first object selected from a plurality of objects; and The point is marked as belonging to a first category among a plurality of categories, wherein each category among the plurality of categories includes at least one corresponding object among the plurality of objects.

Images