Information processing apparatus, information processing method, and information generation method

Abstract

An information processing apparatus according to one embodiment of the present disclosure includes: a three-dimensional point cloud recognition unit that performs a recognition process on a three-dimensional point cloud and provides a recognition result for each point of the three-dimensional point cloud; and an identification confidence calculation unit that calculates a confidence of the identification result for each point of the three-dimensional point cloud, and provides a confidence for each point of the three-dimensional point cloud.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and an information generating method. Background Art

[0002] Currently, regarding 3D recognition such as in autonomous driving, robots, and augmented reality (AR), there are devices that perform recognition processing on 3D point clouds (three-dimensional point clouds) (for example, see Patent Document 1). In such a technology, in the case where it is desired to change the detection strength of the recognition object, the detection strength can be changed by performing recognition processing on the 3D point cloud again after changing the detection strength.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: JP 2019-185347 A Summary of the invention

[0006] Technical issues

[0007] However, generally, recognition processing on 3D data (three-dimensional data) generally has high processing cost, and for example, in a case where it is desired to change the detection intensity of the recognition object, the processing cost increases because recognition processing is required each time.

[0008] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information generating method that can reduce processing costs.

[0009] Technical solution to the problem

[0010] According to one aspect of the present disclosure, an information processing device includes: a three-dimensional point cloud recognition unit, which performs recognition processing on a three-dimensional point cloud and gives a recognition result for each point of the three-dimensional point cloud; and a recognition confidence calculation unit, which calculates the confidence of the recognition result for each point of the three-dimensional point cloud and gives the confidence for each point of the three-dimensional point cloud.

[0011] According to one aspect of the present disclosure, an information processing method includes: performing recognition processing on a three-dimensional point cloud and giving a recognition result for each point of the three-dimensional point cloud; and calculating the confidence of the recognition result for each point of the three-dimensional point cloud and giving the confidence for each point of the three-dimensional point cloud.

[0012] According to one aspect of the present disclosure, an information generating method includes generating information including position information of each point of a three-dimensional point cloud, a recognition result of each point of the three-dimensional point cloud, and a confidence level of the recognition result of each point of the three-dimensional point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram showing an example of a schematic configuration of an information processing system according to the first embodiment.

[0014] Figure 2 is a diagram illustrating an example of a schematic configuration of an information processing apparatus according to the first embodiment.

[0015] Figure 3 : is a diagram for describing a first calculation example of the confidence level of the recognition result of the 3D point cloud according to the first embodiment.

[0016] Figure 4 : is a diagram for describing a second calculation example of the confidence level of the recognition result of the 3D point cloud according to the first embodiment.

[0017] Figure 5 is a diagram for describing a first specific example of the data format of a 3D point cloud according to the first embodiment.

[0018] Figure 6 : is a diagram for describing a second specific example of the data format of the 3D point cloud according to the first embodiment.

[0019] Figure 7 is a flowchart showing an example of the flow of information processing according to the first embodiment.

[0020] Figure 8 : is a diagram for describing assignment of identification labels to each 3D point cloud according to the first embodiment.

[0021] Fig. 9 is a diagram for describing an example of the flow of information processing performed by the information processing apparatus according to the first embodiment.

[0022] Fig.10 is a diagram for describing an example of the flow of information processing performed by an information processing apparatus according to a comparative example of the first embodiment.

[0023] Fig.11 is a diagram illustrating an example of a schematic configuration of an information processing apparatus according to the second embodiment.

[0024] Fig.12 is a diagram for describing integration of a plurality of 2D recognition results corresponding to points of a 3D point cloud according to the second embodiment.

[0025] Fig.13 is a flowchart showing an example of the flow of information processing according to the second embodiment.

[0026] Fig.14 : is a diagram for describing assignment of identification labels to each 3D point cloud according to the second embodiment.

[0027] Fig.15 is a diagram for describing a first specific example of a series of data formats of a 3D point cloud according to various embodiments.

[0028] Fig.16 is a diagram for describing a second specific example of a series of data formats of a 3D point cloud according to various embodiments.

[0029] Fig.17 is a diagram for describing a third specific example of a series of data formats of a 3D point cloud according to various embodiments.

[0030] Fig.18 is a diagram for describing a fourth specific example of a series of data formats of a 3D point cloud according to various embodiments.

[0031] Fig.19 is a diagram for describing a first specific example of a graphical user interface (GUI) according to various embodiments.

[0032] Fig. 20 is a diagram for describing a second specific example of the GUI according to various embodiments.

[0033] Fig.21 is a diagram illustrating an example of a schematic configuration of a first modification example of the information processing device according to each embodiment.

[0034] Fig. 22 is a diagram illustrating an example of a schematic configuration of a second modification example of the information processing device according to each embodiment.

[0035] Fig.23 is a diagram illustrating an example of a schematic configuration of hardware according to various embodiments. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the apparatus, system, method, etc. according to the present disclosure are not limited by the embodiments. In addition, in the following embodiments, the same components are basically represented by the same figure numerals, and redundant descriptions will be omitted.

[0037] One or more embodiments described below (including examples and variations) can be implemented independently of each other. On the other hand, at least part of the multiple embodiments described below can be appropriately combined with at least some of the other embodiments. The multiple embodiments may include novel features that are different from each other. Therefore, the multiple embodiments may help solve different purposes or problems and may show different effects.

[0038] The present disclosure will be described according to the following item order.

[0039] 1. First Implementation Method

[0040] 1-1. Configuration example of information processing system

[0041] 1-2. Configuration example of information processing device

[0042] 1-3. Example of calculating the confidence of a 3D point cloud

[0043] 1-4. Specific example of 3D point cloud data format

[0044] 1-5. Examples of information processing

[0045] 1-6. Differences between the present embodiment and the comparative example

[0046] 1-7. Actions and effects

[0047] 2. Second Implementation

[0048] 2-1. Configuration example of information processing device

[0049] 2-2. Examples of information processing

[0050] 2-3. Actions and effects

[0051] 3. Specific examples of a series of data formats for 3D point clouds

[0052] 4. Specific examples of GUI

[0053] 5. Another Configuration Example of Information Processing Device

[0054] 6. Other Implementation Methods

[0055] 7. Hardware Configuration Example

[0056] 8. Appendix

[0057] <1. First Embodiment>

[0058] <1-1. Configuration example of information processing system>

[0059] Reference Figure 1 A configuration example of the information processing system 1 according to the present embodiment is described. Figure 1 : is a diagram showing an example of a schematic configuration of an information processing system 1 according to the present embodiment.

[0060] like Figure 1 As shown, the information processing system 1 according to the present embodiment includes an information acquisition device 10 , an information processing device 20 , a server device 30 , and an application execution device (application program execution device) 40 .

[0061] For example, the information acquisition device 10 is a device that acquires distance information related to the distance to the object, image information of the object, etc., and sends the acquired distance information and image information to the information processing device 20. The information acquisition device 10 is implemented by a ranging device such as a stereo camera, structured light, a direct time of flight (dToF) sensor, an indirect time of flight (iToF) sensor, and light detection and ranging or laser imaging detection and ranging (LiDAR). However, the information acquisition device 10 may be a device other than a ranging device, and may be any device as long as it can obtain the information from which the information processing device 20 can generate a 3D point cloud (three-dimensional point cloud).

[0062] A stereo camera measures the distance to an object (information in the depth direction of the object) by photographing the object from multiple different directions with a camera. Structured light applies a dot-shaped, stripe-shaped, or grid-shaped pattern light to an object, and photographs the object with a camera at another angle. Since the photographed pattern is distorted according to the shape of the object, information about the shape and depth direction of the object can be obtained based on the distortion. The dToF sensor irradiates an object with a pulsed laser, and measures the distance to the object based on the time difference until the reflected light from the object is detected. The iToF sensor irradiates an object with a periodic laser (continuous wave), and measures the distance to the object based on the phase shift of the reflected light from the object. LiDAR irradiates an object with a laser, measures the time until the light bounces back from the object, and acquires the distance to the object and the shape of the object. Examples of LiDAR include LiDAR using the dToF method and LiDAR using the iToF method. For example, LiDAR includes an irradiation unit that scans and irradiates laser light, a light receiving unit that receives laser light, and the like. In this LiDAR, distance measurement is performed at each angle in the scanning range of the laser in space.

[0063] The information processing device 20 generates a 3D point cloud based on the distance information and image information sent from the information acquisition device 10, gives a recognition result (e.g., a recognition tag indicating the recognition result) for each point of the 3D point cloud, and calculates the confidence of the recognition result of each point. In addition, the information processing device 20 extracts a point cloud with a confidence within a predetermined range from the 3D point cloud based on the confidence of the recognition result of each point of the 3D point cloud, and sends point cloud information related to the point cloud with a confidence within a predetermined range to the server device 30. It should be noted that, for example, a 3D point cloud is a sample of the position and spatial structure of an object (object). Typically, 3D point cloud data is acquired at each frame time of a constant period. By performing various types of calculation processing on the 3D point cloud data, the accurate position, posture, etc. of the object can be detected (recognized).

[0064] For example, the information processing device 20 is implemented by a processor such as a central processing unit (CPU) or a microprocessing unit (MPU). For example, the information processing device 20 is implemented by a processor that executes various programs using a random access memory (RAM) or the like as a work area. It should be noted that the information processing device 20 can be implemented by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Any one of the CPU, MPU, ASIC, and FPGA can be regarded as a controller. In addition, in addition to or instead of the CPU, the information processing device 20 can be implemented by a graphics processing unit (GPU). In addition, the information processing device 20 can be implemented by specific software instead of specific hardware.

[0065] The server device 30 stores and manages various types of information. For example, the server device 30 stores and manages point cloud information transmitted from the information processing device 20. For example, the server device 30 is implemented by a server such as a cloud server, a PC server, a mid-size server, or a mainframe server.

[0066] Here, the sending and receiving of data with the server device 30 is performed via, for example, a network. The network is, for example, a communication network (communication link), such as a local area network (LAN), a wide area network (WAN), a cellular network, a fixed-line link, a regional Internet Protocol (IP) link, or the Internet. The network may include a wired network or a wireless network. In addition, the network may include a core network. The core network is, for example, an evolved packet core (EPC) or a 5G core network (5GC). In addition, the network may include a data network other than the core network. For example, the data network may be a service network of a telecommunications operator, such as an IP Multimedia Subsystem (IMS) network. In addition, the data network may also be a private network, such as an internal company network.

[0067] Note that as a radio access technology (RAT), Long Term Evolution (LTE), New Radio (NR), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. can be used. Several types of radio access technologies can be used, for example, NR and Wi-Fi can be used, and LTE and NR can be used. LTE and NR are types of cellular communication technologies, and mobile communication is achieved by arranging a plurality of areas covered by base stations in a cell shape.

[0068] The application execution device 40 executes the application using the point cloud information transmitted from the server device 30. As the application, for example, various applications such as a general-purpose application and a dedicated application can be used.

[0069] For example, similar to the information processing device 20, the application execution device 40 is implemented by a processor such as a CPU or an MPU. For example, the application execution device 40 is implemented by a processor that executes various programs using a RAM or the like as a work area. It should be noted that the application execution device 40 may also be implemented by an integrated circuit such as an ASIC or an FPGA. In addition, in addition to or instead of a CPU, the application execution device 40 may be implemented by a GPU. In addition, the application execution device 40 may be implemented by specific software rather than specific hardware.

[0070] Here, the application execution device 40 is installed on various devices such as a car, a robot, and a user terminal. However, in addition to the application execution device 40, one or both of the information acquisition device 10 and the information processing device 20 may be appropriately installed on various devices. The user terminal is a terminal used by a user, and receives point cloud information from the server device 30, for example. For example, the user terminal is implemented by a terminal such as a personal computer (e.g., a notebook computer or a desktop computer), a smart device (e.g., a smart phone or a tablet), or a personal digital assistant (PDA). In addition, the user terminal may be implemented by, for example, an xR device (such as an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device). The xR device may be a glasses-type device (e.g., AR / MR / VR glasses) or a head-mounted or goggle-type device (e.g., an AR / MR / VR head-mounted device, an AR / MR / VR goggles). These xR devices may display video for only one eye or may display video for both eyes.

[0071] Note that the application execution device 40 receives the point cloud information from the information processing device 20 via the server device 30, but is not limited to this, and for example, the point cloud information may be received directly from the information processing device 20 without passing through the server device 30. This is appropriately selected according to the configuration, purpose, etc. of the various devices on which the application execution device 40 is installed.

[0072] <1-2. Configuration Example of Information Processing Device>

[0073] Reference Figure 2 A configuration example of the information processing device 20 according to the present embodiment is described. Figure 2 2 is a diagram showing an example of a schematic configuration of the information processing device 20 according to the present embodiment.

[0074] like Figure 2 As shown, the information processing device 20 includes a 3D point cloud generation unit (three-dimensional point cloud generation unit) 21, a 3D point cloud recognition unit (three-dimensional point cloud recognition unit) 22, a 3D point cloud recognition confidence calculation unit (recognition confidence calculation unit) 23 and a 3D point cloud extraction unit (three-dimensional point cloud extraction unit) 24.

[0075] The 3D point cloud generation unit 21 generates a 3D point cloud based on information such as distance information and image information transmitted from the information acquisition device 10. The 3D point cloud is a collection of points having position information (eg, three-dimensional coordinates X, Y, Z).

[0076] The 3D point cloud recognition unit 22 performs recognition processing on the 3D point cloud generated by the 3D point cloud generation unit 21, and gives a recognition result for each point of the 3D point cloud. For example, the 3D point cloud recognition unit 22 performs recognition processing of identifying the attribute of the point for each point of the 3D point cloud, and gives a recognition result (e.g., recognition label). The recognition result is a result of recognizing what the object is, and indicates, for example, a car, a person, a building, etc. Note that the recognition processing can be performed, for example, based on a model learned by a neural network (e.g., CNN: Convolutional Neural Network, etc.) as an example of machine learning, or can be performed based on another method.

[0077] The 3D point cloud recognition confidence calculation unit 23 calculates the confidence of the recognition result for each point of the 3D point cloud by the 3D point cloud recognition unit 22, and gives the confidence for each point of the 3D point cloud. The confidence indicates the possibility of the recognition result and is set in a range of, for example, 0.0 to 1.0. For example, the higher the value, the higher the confidence. As an example, if the recognition result indicates a car and the confidence of the recognition result is 0.3, the probability that the confidence indicates that the object is a car is 0.3.

[0078] The 3D point cloud extraction unit 24 extracts some point clouds from all 3D point clouds based on the confidence of each point of the 3D point cloud obtained by the 3D point cloud recognition confidence calculation unit 23. For example, the 3D point cloud extraction unit 24 extracts point clouds with confidence within a predetermined range from the 3D point cloud based on the confidence of each point of the 3D point cloud. The predetermined range is pre-set by a user or the like. For example, when the predetermined range is set to be greater than 0.9, point clouds with confidence of greater than 0.9 are extracted, and when the predetermined range is set to be greater than 0.3 and less than 0.5, point clouds with confidence of greater than 0.3 and less than 0.5 are extracted.

[0079] Note that each block constituting the above-mentioned information processing device 20 (for example, the 3D point cloud generation unit 21, the 3D point cloud recognition unit 22, the 3D point cloud recognition confidence calculation unit 23, and the 3D point cloud extraction unit 24) is a functional block indicating the function of the information processing device 20. These functional blocks may be software blocks or hardware blocks. For example, each block may be a software module implemented by software (microprogram) or a circuit block on a semiconductor chip (bare die). Of course, each block may be a processor or an integrated circuit. In addition, the information processing device 20 may include functional units different from the above-mentioned blocks. The configuration method of each block is arbitrary. In addition, part or all of the operations of each block may be performed by another device.

[0080] <1-3. Example of calculating the confidence level of a 3D point cloud>

[0081] Reference Figure 3 and Figure 4 An example of calculating the confidence level of a 3D point cloud according to the present embodiment will be described. Figure 3 and Figure 4 is a diagram for describing an example of calculation of the confidence level of a 3D point cloud according to the present embodiment. Figure 3 and Figure 4 A, B, C, and D in indicate the types of identification tags, and indicate that the identification results are different.

[0082] like Figure 3 As shown in the first calculation example, the 3D point cloud recognition confidence calculation unit 23 calculates the confidence of the focus point R1 based on the identification tags of the neighboring points of the focus point R1. The neighboring points are, for example, points existing in a spherical area R2 with a predetermined radius centered on the focus point R1. Figure 3 In the example of , there are four points B, one point A, and one point C in the spherical region R2. At this time, the number of points having the same identification label as B of the focus point R1 is four, and the number of points within the spherical region R2 is six. In this example, for the identification label B, the confidence of the focus point R1 is calculated as 4 / 6.

[0083] like Figure 4 As shown in the second calculation example, the 3D point cloud recognition confidence calculation unit 23 calculates the confidence of the focus point R1 based on the neighboring points and the color of the focus point R1 (i.e., based on the identification labels of the neighboring points of the focus point R1, while also considering the color of the points). As described above, the neighboring points are, for example, points that exist within a spherical area R2 with a predetermined radius centered on the focus point R1. Figure 4 In the example of FIG. 1 , there are seven points including the focus point R1 whose colors are close to the color of the focus point R1 in the spherical region R2 (see FIG. 1 ). Figure 4). The number of points with the same identification label as B of the focus point R1 is five. In this example, for the identification label B, the confidence of the focus point R1 is calculated as 5 / 7.

[0084] According to the first calculation example or the second calculation example, the 3D point cloud recognition confidence calculation unit 23 may calculate the statistics of the local area (e.g., the spherical area R2) based on the positional relationship and color similarity of each point in the local area of ​​the 3D point cloud, and calculate the confidence of each point of the 3D point cloud based on the statistics. Figure 3 and Figure 4 The calculation example of is only an example, and other confidence calculation methods may also be used. Although the spherical region R2 is used as the local region containing the focus point R1, the present invention is not limited thereto, and a spatial region having another shape may also be used.

[0085] <1-4. Specific example of 3D point cloud data format>

[0086] Reference Figure 5 and Figure 6 A specific example of the data format of the 3D point cloud according to the present embodiment is described. Figure 5 and Figure 6 is a diagram for describing a specific example of the data format of a 3D point cloud according to the present embodiment.

[0087] like Figure 5 As shown in the first specific example, the 3D point cloud recognition confidence calculation unit 23 adds an identification label and a confidence corresponding to the identification label to each point of the 3D point cloud. Figure 5 In the example, in addition to X, Y, Z (position information) and R, G, B (color information), labels (identification labels) and confidence information are added to each point of the 3D point cloud.

[0088] like Figure 6 As shown in the second specific example, the 3D point cloud recognition confidence calculation unit 23 adds multiple recognition tags and confidences corresponding to each recognition tag to each point of the 3D point cloud. Figure 6 In the example, in addition to X, Y, Z (position information) and R, G, B (color information), multiple labels (identification labels) and confidence information of each label are added to each point in the 3D point cloud.

[0089] According to the first specific example or the second specific example, in addition to the position information and color information of the 3D point cloud data, information of the identification label and the confidence level or information of multiple identification labels and the confidence levels of the respective identification labels are added. Figure 5 and Figure 6The specific examples of are merely examples, and other data formats may also be used.

[0090] Here, the 3D point cloud recognition confidence calculation unit 23 generates data including at least the position information of each point of the 3D point cloud, the identification label of each point of the 3D point cloud, and the confidence of the identification label of each point of the 3D point cloud, and sends the data to the 3D point cloud extraction unit 24. For example, the 3D point cloud extraction unit 24 stores the data sent from the 3D point cloud recognition confidence calculation unit 23 in a storage unit. The storage unit may be provided in the 3D point cloud extraction unit 24 or may be provided outside the 3D point cloud extraction unit 24. The storage unit is implemented by, for example, a storage device capable of reading and writing data (such as DRAM, SRAM, flash memory, or hard disk).

[0091] <1-5. Examples of information processing>

[0092] Reference Figure 7 and Figure 8 An example of information processing according to the present embodiment is described. Figure 7 is a flowchart showing an example of the flow of information processing according to the present embodiment. Figure 8 3D is a diagram for describing the assignment of identification labels to each 3D point cloud according to the present embodiment.

[0093] like Figure 7 As shown, in step S11, the information acquisition device 10 performs sensing and acquires sensing information (e.g., distance information, etc.). In step S12, the 3D point cloud generation unit 21 generates a 3D point cloud based on the sensing information. In step S13, the 3D point cloud recognition unit 22 performs recognition processing on the 3D point cloud and calculates the recognition result of each point. In step S14, the 3D point cloud recognition confidence calculation unit 23 calculates the confidence of the recognition result of each point of the 3D point cloud. In step S15, the 3D point cloud extraction unit 24 extracts a point cloud from the 3D point cloud using the confidence of the recognition result of each point of the 3D point cloud. For example, the 3D point cloud extraction unit 24 extracts a point cloud with a confidence within a predetermined range from the 3D point cloud.

[0094] Here, in step S13, Figure 8 As shown, recognition processing is performed on the 3D point cloud, and a recognition result (recognition label) is calculated and given for each point of the 3D point cloud. Figure 8 In the example of , there are various fill areas, for example, a black solid fill area and a dot fill area, and these different fill areas indicate different recognition results.

[0095] <1-6. Differences between the present embodiment and the comparative example>

[0096] Reference Fig. 9 and Fig.10 The difference between the present embodiment according to the present embodiment and the comparative example is described. Fig. 9 2 is a diagram for describing an example of the flow of information processing performed by the information processing device 20 according to the present embodiment. Fig.10 : is a diagram for describing an example of the flow of information processing performed by the information processing device 50 according to the comparative example of the present embodiment.

[0097] like Fig. 9 As shown, the information processing device 20 according to the present embodiment includes the 3D point cloud generation unit 21, the 3D point cloud recognition unit 22, the 3D point cloud recognition confidence calculation unit 23 and the 3D point cloud extraction unit 24 as described above. Fig. 9 In the example, in the first execution (execution of the process), the desired detection strength for detecting the identification object is set, and the generation of the 3D point cloud, the assignment of the identification label to each point, the confidence calculation of the identification result of each point, and the point cloud extraction with the confidence within a predetermined range are performed. In the second and subsequent executions, other detection strengths are set, and only the point cloud extraction with the confidence within the predetermined range is performed. The detection strength corresponds to the confidence. According to such processing, by calculating the confidence outside the recognition processing of the 3D point cloud, even if the detection strength is changed in the recognition processing of the 3D point cloud, the recognition result with the changed detection strength can be obtained without re-executing the recognition processing. Therefore, the change of the detection strength for identifying the 3D point cloud can be achieved at a low cost in the subsequent stage.

[0098] Here, for example, the setting of the detection strength is performed by an input operation of a user on a device (e.g., a vehicle, a robot, a user terminal, etc.) on which the application execution device 40 is installed. For example, the setting of the detection strength is changed according to a purpose such as trial and error. In response to the change in the detection strength, the 3D point cloud extraction unit 24 extracts a point cloud with a confidence within a predetermined range, for example, a point cloud with a confidence equal to or higher than the detection strength, based on the confidence of each point of the 3D point cloud calculated once. As a result, the recognition result of a 3D point cloud of arbitrary detection strength can be obtained at a low cost.

[0099] Note that a point cloud having a confidence level equal to or higher than the set detection strength is extracted, but the present invention is not limited thereto. For example, a point cloud having only a confidence level corresponding to the set detection strength may be extracted. In addition to the change in detection strength, for example, if a detection object (recognition object) can be selected in the configuration, the detection object may be changed. In this case, in response to the change in the detection object, the 3D point cloud extraction unit 24 extracts a point cloud having a confidence level within a predetermined range corresponding to the detection object in the 3D point cloud based on the confidence level of each point having a recognition tag corresponding to the detection object.

[0100] On the other hand, Fig.10 As shown, the information processing device 50 according to the comparative example includes a 3D point cloud generation unit 51, a 3D point cloud recognition unit 52, and a 3D point cloud extraction unit 53. The 3D point cloud generation unit 51 performs processing similar to the above-mentioned 3D point cloud generation unit 21. The 3D point cloud recognition unit 52 performs recognition processing for each point of the 3D point cloud. The 3D point cloud extraction unit 53 performs point cloud extraction based on the recognition result. Fig.10 In the example of , in the first execution (execution of the process), the desired detection strength is set, and the generation of the 3D point cloud, the recognition of each point, and the point cloud extraction based on the recognition result are performed. In the second and subsequent executions, other detection strengths are set, and the recognition for each point and the point cloud extraction based on the recognition result are performed. According to such a process, when the detection strength is changed in the recognition process of the 3D point cloud, it is necessary to re-execute the recognition process each time, and the processing cost is high.

[0101] <1-7. Actions and Effects>

[0102] As described above, according to the present embodiment, the information processing device 20 includes: a 3D point cloud recognition unit 22 that performs recognition processing on a 3D point cloud and gives a recognition result (e.g., a recognition label) for each point of the 3D point cloud; and a 3D point cloud recognition confidence calculation unit 23 that calculates the confidence of the recognition result for each point of the 3D point cloud and gives the confidence for each point of the 3D point cloud. As a result, since the confidence is given for each point of the 3D point cloud, the point cloud can be extracted according to the confidence of each point of the 3D point cloud using the confidence of each point of the 3D point cloud. Therefore, for example, in the case where it is desired to change the detection strength of the recognition processing, it is only necessary to perform point cloud extraction according to the confidence of each point of the 3D point cloud without re-execution of the recognition processing that is conventionally necessary, and the processing cost can be suppressed. In addition, a point cloud with a confidence within an arbitrary range can also be extracted.

[0103] In addition, the 3D point cloud recognition confidence calculation unit 23 can calculate the confidence of each point of the 3D point cloud based on the statistical data of the local area (e.g., spherical area R2) of the 3D space of the recognition result of each point of the 3D point cloud. As a result, the confidence of each point of the 3D point cloud can be accurately obtained.

[0104] In addition, the 3D point cloud recognition confidence calculation unit 23 can calculate the statistics of the local area based on the positional relationship of each point in the local area of ​​the 3D point cloud. As a result, the confidence of each point of the 3D point cloud can be obtained more accurately.

[0105] In addition, the 3D point cloud recognition confidence calculation unit 23 can calculate the statistics of the local area based on the similarity of the color of each point in the local area of ​​the 3D point cloud. As a result, the confidence of each point of the 3D point cloud can be obtained more accurately.

[0106] In addition, the information processing device 20 may further include a 3D point cloud extraction unit 24 that extracts a point cloud from the 3D point cloud based on the confidence of each point of the 3D point cloud. As a result, a point cloud can be obtained from the 3D point cloud based on the confidence of each point of the 3D point cloud.

[0107] In addition, the 3D point cloud extraction unit 24 can extract a point cloud having a confidence within a predetermined range from the 3D point cloud based on the confidence of each point of the 3D point cloud. As a result, a point cloud having a confidence within a predetermined range can be obtained from the 3D point cloud.

[0108] In addition, the 3D point cloud extraction unit 24 can store the confidence of each point of the 3D point cloud, and extract the point cloud from the 3D point cloud based on the stored confidence of each point of the 3D point cloud. As a result, the point cloud can be reliably obtained from the 3D point cloud.

[0109] In addition, the information processing device 20 may further include a 3D point cloud generation unit 21 that generates a 3D point cloud. As a result, a 3D point cloud can be reliably obtained.

[0110] In addition, the 3D point cloud generation unit 21 can generate a 3D point cloud based on the 3D distance information. As a result, a correct 3D point cloud can be obtained.

[0111] <2. Second Embodiment>

[0112] <2-1. Configuration example of information processing device>

[0113] Reference Fig.11 A configuration example of the information processing device 20 according to the present embodiment is described. Fig.11 2 is a diagram showing an example of a schematic configuration of the information processing device 20 according to the present embodiment. The present embodiment is basically the same as the first embodiment, but the difference thereof (regarding the configuration and processing of a 2D image group) will be described.

[0114] like Fig.11As shown, for example, the information acquisition device 10 according to the present embodiment acquires image information, distance information, etc. related to a 2D image (two-dimensional image) of an object, and transmits the acquired image information, distance information, etc. to the information processing device 20. As the 2D image of the object, for example, a group of two or more 2D images captured with overlapping areas are acquired. Note that the transmission of the image information can be realized by, for example, a radio access technology such as Wi-Fi (registered trademark).

[0115] The information processing device 20 includes a 2D recognition unit (two-dimensional recognition unit) 25 and a 3D point cloud recognition integration unit (three-dimensional point cloud recognition integration unit) 26 instead of the 3D point cloud recognition unit 22 according to the first embodiment (see Figure 2 ). The 2D recognition unit 25 and the 3D point cloud recognition integration unit 26 serve as a 3D point cloud recognition unit.

[0116] The 3D point cloud generation unit 21 generates a 3D point cloud from the 2D image group based on the 2D image group, distance information, etc. obtained by the information acquisition device 10. For example, structure from motion (SfM) and multi-view stereo vision (MVS) are used as a method for generating a 3D point cloud from the 2D image group. SfM generates low-density point cloud data restored from feature points by triangulation. MVS generates high-density point cloud data.

[0117] The 2D recognition unit 25 performs recognition processing on a plurality of 2D images, and gives a recognition result for each image element. For example, the 2D recognition unit 25 performs recognition processing for recognizing the attribute of the image element for each image element, and gives a label (2D recognition label) indicating the 2D recognition result. The 2D recognition result is a recognition result of the image element of the 2D image. Note that the recognition processing may be performed, for example, based on a model learned by a neural network (e.g., CNN: Convolutional Neural Network, etc.) as an example of machine learning, or may be performed based on another method.

[0118] The 3D point cloud recognition integration unit 26 reflects the recognition result of each 2D image in the 3D point cloud. For example, the 3D point cloud recognition integration unit 26 integrates a plurality of 2D recognition results corresponding to the points of the 3D point cloud, and gives a label (3D recognition label) indicating the 3D recognition result to each point of the 3D point cloud. The 3D recognition result is the recognition result of the points of the 3D point cloud. Note that, for example, the integration is performed by majority decision. Because a plurality of 2D recognition results are integrated into the recognition result of the 3D point cloud, the accuracy of the recognition result can be improved.

[0119] Here, Fig.12 is a diagram for describing the integration of a plurality of 2D recognition results corresponding to points of a 3D point cloud according to the present embodiment. Fig.12As shown, each 2D image group has a recognition result, that is, a 2D recognition label for each image element, and the 3D point cloud has multiple 2D recognition labels corresponding to each point. In this case, for example, the 2D recognition label with the largest number of points of interest for the 3D point cloud is adopted as the 3D recognition label. As an example, if the multiple 2D recognition labels corresponding to the point of interest are building, building, building and car, the 3D recognition label for the point of interest is building.

[0120] return Fig.11 , the 3D point cloud recognition confidence calculation unit 23 calculates the confidence of the recognition result for each point of the 3D point cloud, and gives the confidence for each point of the 3D point cloud. For example, the 3D point cloud recognition confidence calculation unit 23 calculates the confidence based on the statistical data of multiple 2D recognition results corresponding to the points (3D points) of the 3D point cloud. For example, assuming that the confidence is Conf, num_label_X is the number of 2D recognition results X corresponding to the 3D point, and N is the number of 2D points (2D recognition results) corresponding to the 3D point, the confidence is obtained according to Conf=num_label_X / N.

[0121] As in the first embodiment, the 3D point cloud extraction unit 24 extracts some point clouds from all 3D point clouds based on the confidence of each point of the 3D point cloud obtained by the 3D point cloud recognition confidence calculation unit 23. For example, the 3D point cloud extraction unit 24 extracts point clouds having a confidence within a predetermined range from the 3D point clouds based on the confidence of each point of the 3D point cloud.

[0122] Here, each block (3D point cloud generation unit 21, 2D recognition unit 25, 3D point cloud recognition integration unit 26, 3D point cloud recognition confidence calculation unit 23 and 3D point cloud extraction unit 24) constituting the above-mentioned information processing device 20 is a functional block indicating the function of the information processing device 20, as in the first embodiment. These functional blocks can be software blocks or hardware blocks. For example, each block can be a software module implemented by software (microprogram) or a circuit block on a semiconductor chip (bare die). Of course, each block can be a processor or an integrated circuit. In addition, the information processing device 20 may include functional units different from the above-mentioned blocks. The configuration method of each block is arbitrary. In addition, part or all of the operations of each block can be performed by another device.

[0123] <2-2. Examples of information processing>

[0124] Will refer to Fig.13 and Fig.14 An example of information processing according to the present embodiment is described. Fig.13 is a flowchart showing an example of the flow of information processing according to the present embodiment. Fig.143D is a diagram for describing the assignment of identification labels to each 3D point cloud according to the present embodiment.

[0125] like Fig.13 As shown, in step S21, the information acquisition device 10 continuously performs shooting in a manner that there are overlapping areas, and acquires multiple color images. In step S22, the 3D point cloud generation unit 21 generates a 3D point cloud based on each color image. In step S23, the 2D recognition unit 25 performs recognition processing for each image element of each color image, and calculates the recognition result of each image element of each color image. In step S24, the 3D point cloud recognition integration unit 26 uses the correspondence between the 3D point cloud and the color image to calculate the recognition result of each point of the 3D point cloud. In step S25, the 3D point cloud recognition confidence calculation unit 23 calculates the confidence of the recognition result of each point of the 3D point cloud. In step S26, the 3D point cloud extraction unit 24 extracts a point cloud from the 3D point cloud using the confidence of the recognition result of each point of the 3D point cloud. For example, the 3D point cloud extraction unit 24 extracts a point cloud with a confidence within a predetermined range from the 3D point cloud.

[0126] like Fig.14 As shown, in step S23, recognition processing is performed on a group of images captured to have overlapping areas (ie, more than two 2D images). As a result, a recognition label map is generated. Fig.14 In the example of , the 2D recognition tags are "soil", "plant", "asphalt", "water", "construction equipment", "building", "object", "person", and "car". In this case, for example, the information acquisition device 10 is installed on a mobile body such as a drone, an airplane, or a helicopter equipped with an aerial camera. Next, in step S24, the 2D recognition result (for example, the 2D recognition tag) is reflected in the 3D point cloud. It should be noted that in Fig.14 In the example of , there are various fill areas such as a black solid fill area and a dot fill area, and these different fill areas indicate that the recognition results are different.

[0127] <2-3. Actions and Effects>

[0128] As described above, in the present embodiment, the effect according to the first embodiment can be obtained. That is, according to the second embodiment, the information processing device 20 includes a 2D recognition unit 25 that performs recognition processing on a plurality of 2D images and a 3D point cloud recognition integration unit 26 that reflects the recognition result of each 2D image in a 3D point cloud. As a result, by performing recognition processing on a plurality of 2D images, reflecting the recognition result of each 2D image in a 3D point cloud, and providing confidence for each point of the 3D point cloud, the processing cost can be suppressed as in the first embodiment.

[0129] In addition, when the recognition result of each 2D image is reflected in the 3D point cloud, the 3D point cloud recognition confidence calculation unit 23 can calculate the confidence of each point of the 3D point cloud according to the statistical data of the recognition result of each 2D image corresponding to the 3D point cloud. As a result, the confidence of each point of the 3D point cloud can be accurately obtained.

[0130] In addition, the 3D point cloud generation unit 21 may generate a 3D point cloud based on a plurality of 2D images. As a result, a 3D point cloud may be obtained using a plurality of 2D images.

[0131] <3. Specific examples of a series of data formats for 3D point clouds>

[0132] Will refer to Figures 15 to 18 A series of specific examples of data formats of 3D point clouds according to various embodiments are described. Fig.15 and Fig.18 is a diagram for describing a series of specific examples of data formats of 3D point clouds according to various embodiments.

[0133] like Fig.15 As shown, in the first specific example, the data output from the 3D point cloud generation unit 21 includes X, Y, Z, R, G, and B. XYZ represents position information (three-dimensional position information) in a three-dimensional coordinate system, and RGB represents color information. The data output from the 3D point cloud recognition unit 22 includes X, Y, Z, R, G, B, and a label. The label indicates an identification label as a recognition result. The data output from the 3D point cloud recognition confidence calculation unit 23 includes X, Y, Z, R, G, B, a label, and confidence. The confidence indicates the confidence of the recognition result. The data output from the 3D point cloud extraction unit 24 includes X, Y, Z, R, G, and B. The data is data related to a point cloud having a confidence within a predetermined range.

[0134] like Fig.16 As shown in the second specific example, the data output from the 2D recognition unit 25 includes a label map group. The label map is a map indicating the label of each image element. The data output from the 3D point cloud recognition integration unit 26 includes the 3D point cloud recognition confidence calculation unit 23 (see Fig.15 ) in the X, Y, Z, R, G, B, and labels. Note that the data output from other units is the same as Fig.15 The data in are the same.

[0135] like Fig.17 As shown in the third specific example, the data output from the 3D point cloud generation unit 21 includes X, Y, Z, R, G, and B. The data output from the 3D point cloud recognition unit 22 includes X, Y, Z, and a label. The data output from the 3D point cloud recognition confidence calculation unit 23 does not include R, G, and B (with Fig.15The data output from the 3D point cloud extraction unit 24 includes X, Y, and Z. The data is data related to a point cloud having a confidence level within a predetermined range.

[0136] like Fig.18 As shown in the fourth specific example, the data output from the 2D recognition unit 25 includes a label mapping group. The data output from the 3D point cloud recognition integration unit 26 includes the data as shown in the 3D point cloud recognition confidence calculation unit 23 (see Fig.17 ) in the X, Y, Z, R, G, B, and labels. Note that the data output from other units is the same as Fig.17 The data in are the same.

[0137] According to any one of the first to fourth specific examples, the 3D point cloud recognition confidence calculation unit 23 generates data including at least the position information of each point of the 3D point cloud, the recognition result (e.g., recognition label) of each point of the 3D point cloud, and the confidence of the recognition result of each point of the 3D point cloud. This data is used again by the 3D point cloud extraction unit 24 according to the change of the detection intensity, and is therefore stored in a storage unit or the like.

[0138] <4. Specific example of GUI>

[0139] Will refer to Fig.19 and Fig. 20 Specific examples of GUI (Graphical User Interface) according to various embodiments are described. Fig.19 and Fig. 20 are diagrams for describing specific examples of GUIs according to various embodiments.

[0140] like Fig.19 As shown, the user terminal 100 includes a display unit 110. The display unit 110 is, for example, a liquid crystal display or an organic electroluminescent (EL) display, and a touch panel is used as the display unit 110.

[0141] The display unit 110 includes a display area 111, an object GUI 112, and a detection strength setting GUI 113. The display unit 110 displays a 3D point cloud in the display area 111 based on the point cloud information. The object GUI 112 includes a plurality of check boxes for selecting objects to be extracted. The user checks the check boxes and selects one or more objects. Examples of objects include soil, plants, asphalt, water, construction equipment, buildings, objects, people, cars, etc. The detection strength setting GUI 113 includes a slide bar for specifying the detection strength. For example, the user slides the slide bar to set the detection strength in the range of 0.0 to 1.0.

[0142] The user operates the object GUI 112 and selects an object to be extracted, and operates the detection strength setting GUI 113 to arbitrarily set the detection strength. In response to this, for example, the user terminal 100 transmits information such as the object to be extracted and the detection strength to the information processing device 20 via the server device 30. The 3D point cloud extraction unit 24 receives the transmitted information such as the object to be extracted and the detection strength, extracts a point cloud having a confidence equal to or higher than the detection strength of the identification tag corresponding to the object to be extracted (a point cloud having a confidence within a predetermined range), and transmits point cloud information related to the extracted point cloud to the user terminal 100 via the server device 30. Thereafter, the display unit 110 displays the 3D point cloud in the display area 111 based on the transmitted point cloud information. In this way, the point cloud extracted according to the selection of the object to be extracted and the setting of the detection strength is displayed. For example, when all objects are selected and the detection strength is set to 0.0, all point clouds are displayed.

[0143] It should be noted that, for example, Fig. 20 As shown, the detection strength setting GUI 113 may include a slide bar for specifying a range of detection strengths (a range of objects to be extracted) instead of (or in addition to) a slide bar for specifying detection strengths.

[0144] <5. Another Configuration Example of Information Processing Device>

[0145] Will refer to Fig.21 and Fig. 22 Another configuration example of the information processing device 20 according to the above-described respective embodiments is described. Fig.21 and Fig. 22 is a diagram showing an example of a schematic configuration of a modification of the information processing device 20 according to each embodiment.

[0146] like Fig.21 As shown, the information processing device 20 may include a 3D point cloud generation unit 21, and the server device 30 may include a 3D point cloud recognition unit 22, a 3D point cloud recognition confidence calculation unit 23, and a 3D point cloud extraction unit 24. According to this configuration, the amount of data to be transmitted by the information processing device 20 is six (X, Y, Z, R, G, and B) points per 3D point cloud. The advantage is that heavy recognition processing can be performed on the server device 30 (e.g., cloud server) side with abundant computing resources.

[0147] like Fig. 22As shown, the information processing device 20 may include a 3D point cloud generation unit 21, a 3D point cloud recognition unit 22, a 3D point cloud recognition confidence calculation unit 23, and the server device 30 may include a 3D point cloud extraction unit 24. According to this configuration, the amount of data to be transmitted by the information processing device 20 is five (X, Y, Z, label and confidence) bars per point of the 3D point cloud. The advantage is that color information does not need to be sent to the server device 30 (e.g., cloud server) side.

[0148] <6. Other Implementation Methods>

[0149] The processing according to the above embodiment (or variant) can be performed in various different modes (variations) different from the above embodiment. For example, in the processing described in the above embodiment, all or part of the processing described as automatically performed can be performed manually, or all or part of the processing described as manually performed can be performed automatically by a known method. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be arbitrarily changed. For example, the various types of information shown in each figure are not limited to the information shown.

[0150] In addition, each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific form of distribution and integration of each device is not limited to the form shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0151] In addition, the above-described embodiments (or modified examples) can be appropriately combined within a range that does not contradict the processing contents. In addition, the effects described in this specification are only examples and are not limited, and other effects may be provided.

[0152] <7. Hardware Configuration Example>

[0153] A specific hardware configuration example of information devices such as the information processing device 20, the server device 30, and the application execution device 40 according to the above-mentioned embodiment (or modification example) will be described. Fig.23 The computer 500 of the configuration shown in embodies information equipment such as the information processing device 20 , the server device 30 , and the application execution device 40 according to the present embodiment (or a modified example). Fig.23 is a diagram showing an example of a schematic configuration of hardware that realizes the functions of an information device.

[0154] like Fig.23As shown, the computer 500 includes a CPU 510 , a RAM 520 , a read only memory (ROM) 530 , a hard disk drive (HDD) 540 , a communication interface 550 , and an input / output interface 560 . Each unit of the computer 500 is connected through a bus 570 .

[0155] The CPU 510 operates based on the program stored in the ROM 530 or the HDD 540, and controls each unit. For example, the CPU 510 develops the program stored in the ROM 530 or the HDD 540 in the RAM 520, and executes processing corresponding to the various programs.

[0156] The ROM 530 stores a boot program such as a basic input output system (BIOS) executed by the CPU 510 when the computer 500 is activated, a program depending on the hardware of the computer 500 , and the like.

[0157] The HDD 540 is a recording medium that can be read by the computer 500 and performs non-transitory recording of programs executed by the CPU 510, data used by such programs, etc. Specifically, the HDD 540 is a recording medium that records an information processing program according to the present disclosure as an example of program data 541.

[0158] The communication interface 550 is an interface for connecting the computer 500 to an external network 580 (eg, the Internet). For example, the CPU 510 receives data from another device via the communication interface 550 or transmits data generated by the CPU 510 to another device.

[0159] The input / output interface 560 is an interface for connecting the input / output device 590 and the computer 500. For example, the CPU 510 receives data from an input device such as a keyboard or a mouse via the input / output interface 560. In addition, the CPU 510 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 560.

[0160] It should be noted that, in addition, the input / output interface 560 can be used as a medium interface for reading a program or the like recorded in a predetermined recording medium (medium). For example, the medium is an optical recording medium such as a digital versatile disk (DVD) or a phase-change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, or the like.

[0161] Here, for example, in the case where the computer 500 is used as an information device such as the information processing device 20, the server device 30, or the application execution device 40 according to various embodiments (or modified examples), the CPU 510 of the computer 500 executes the information processing program loaded on the RAM 520 to realize all or part of the functions of the various units such as the information processing device 20, the server device 30, or the application execution device 40 according to various embodiments (or modified examples). In addition, the HDD 540 stores the information processing program and data according to various embodiments. Note that the CPU 510 reads the program data 541 from the HDD 540 and executes the program data, but as another example, these programs may be acquired from another device via the external network 580.

[0162] <8. Appendix>

[0163] It should be noted that the present technology may also have the following configurations.

[0164] (1) An information processing device comprising:

[0165] a three-dimensional point cloud recognition unit that performs recognition processing on the three-dimensional point cloud and gives a recognition result for each point of the three-dimensional point cloud; and

[0166] A recognition confidence calculation unit calculates the confidence of the recognition result for each point of the three-dimensional point cloud and provides the confidence for each point of the three-dimensional point cloud.

[0167] (2) The information processing device according to (1), wherein:

[0168] The recognition confidence calculation unit calculates the confidence of each point of the three-dimensional point cloud according to the statistical data of the local area of ​​the three-dimensional space of the recognition result of each point of the three-dimensional point cloud.

[0169] (3) The information processing device according to (2), wherein:

[0170] The recognition confidence calculation unit calculates statistical data of the local area based on the positional relationship of each point in the local area of ​​the three-dimensional point cloud.

[0171] (4) The information processing device according to (2) or (3), wherein:

[0172] The recognition confidence calculation unit calculates statistics of the local area based on the similarity of the color of each point in the local area of ​​the three-dimensional point cloud.

[0173] (5) The information processing device according to any one of (1) to (4), wherein:

[0174] The 3D point cloud recognition unit includes:

[0175] a two-dimensional recognition unit that performs recognition processing on a plurality of two-dimensional images; and

[0176] A three-dimensional point cloud recognition and integration unit reflects the recognition result of each two-dimensional image in the three-dimensional point cloud.

[0177] (6) The information processing device according to (5), wherein:

[0178] When the recognition result of each two-dimensional image is reflected in the three-dimensional point cloud, the recognition confidence calculation unit calculates the confidence of each point of the three-dimensional point cloud based on the statistical data of the recognition result of each two-dimensional image corresponding to the three-dimensional point cloud.

[0179] (7) The information processing device according to any one of (1) to (6), further comprising:

[0180] A three-dimensional point cloud extraction unit extracts a point cloud from the three-dimensional point cloud based on the confidence level of each point of the three-dimensional point cloud.

[0181] (8) The information processing device according to (7), wherein:

[0182] The three-dimensional point cloud extraction unit extracts a point cloud having a confidence level within a predetermined range from the three-dimensional point cloud based on the confidence level of each point of the three-dimensional point cloud.

[0183] (9) The information processing device according to (7) or (8), wherein:

[0184] The three-dimensional point cloud extraction unit stores the confidence of each point of the three-dimensional point cloud, and extracts a point cloud from the three-dimensional point cloud based on the stored confidence of each point of the three-dimensional point cloud.

[0185] (10) The information processing device according to any one of (1) to (9), further comprising:

[0186] A three-dimensional point cloud generating unit generates a three-dimensional point cloud.

[0187] (11) The information processing device according to (10), wherein:

[0188] The three-dimensional point cloud generating unit generates a three-dimensional point cloud according to the three-dimensional distance information.

[0189] (12) The information processing device according to (10), wherein:

[0190] The three-dimensional point cloud generating unit generates a three-dimensional point cloud according to a plurality of two-dimensional images.

[0191] (13) An information processing method comprising:

[0192] performing recognition processing on the three-dimensional point cloud and giving a recognition result for each point of the three-dimensional point cloud; and

[0193] The confidence of the recognition result is calculated for each point of the three-dimensional point cloud and the confidence is given for each point of the three-dimensional point cloud.

[0194] (14) A method for generating information, comprising:

[0195] Generate information including:

[0196] The position information of each point in the 3D point cloud,

[0197] The recognition results of each point of the 3D point cloud, and

[0198] The confidence level of the recognition result for each point in the 3D point cloud.

[0199] (15) An information processing system including the information processing device according to any one of (1) to (12).

[0200] (16) An information processing method using the information processing device according to any one of (1) to (12).

[0201] (17) An information generating method using the information processing device according to any one of (1) to (12).

[0202] Reference numerals list

[0203] 1 Information Processing System

[0204] 10 Information acquisition device

[0205] 20 Information processing device

[0206] 21 3D point cloud generation unit

[0207] 22 3D point cloud recognition unit

[0208] 23 3D point cloud recognition confidence calculation unit

[0209] 24 3D point cloud extraction unit

[0210] 25 2D recognition unit

[0211] 26 3D point cloud recognition integration unit

[0212] 30 Server Devices

[0213] 40 Application execution device

[0214] 50 Information processing device

[0215] 51 3D point cloud generation unit

[0216] 52 3D point cloud recognition unit

[0217] 53 3D point cloud extraction unit

[0218] 100 User Terminals

[0219] 110 Display unit

[0220] 111 Display area

[0221] 112 Object GUI

[0222] 113 Detection intensity setting GUI

[0223] 500 Computer

[0224] R1 Focus

[0225] R2 Spherical region.

Claims

1. An information processing device, comprising: a three-dimensional point cloud recognition unit, which performs recognition processing on the three-dimensional point cloud and provides a recognition result for each point of the three-dimensional point cloud; as well as A recognition confidence calculation unit is configured to calculate the confidence of the recognition result for each point of the three-dimensional point cloud and to provide the confidence for each point of the three-dimensional point cloud.

2. The information processing device according to claim 1, wherein: The recognition confidence calculation unit calculates the confidence of each point of the three-dimensional point cloud according to the statistical data of the local area of ​​the three-dimensional space of the recognition result of each point of the three-dimensional point cloud.

3. The information processing device according to claim 2, wherein: The recognition confidence calculation unit calculates the statistical data of the local area based on a positional relationship of each point in the local area of ​​the three-dimensional point cloud.

4. The information processing device according to claim 2, wherein: The recognition confidence calculation unit calculates the statistical data of the local area based on the similarity of the color of each point in the local area of ​​the three-dimensional point cloud.

5. The information processing device according to claim 1, wherein: The three-dimensional point cloud recognition unit comprises: a two-dimensional recognition unit that performs recognition processing on a plurality of two-dimensional images; and A three-dimensional point cloud recognition and integration unit is provided, wherein the three-dimensional point cloud recognition and integration unit reflects the recognition result of each of the two-dimensional images in the three-dimensional point cloud.

6. The information processing device according to claim 5, wherein: When the recognition result of each of the two-dimensional images is reflected in the three-dimensional point cloud, the recognition confidence calculation unit calculates the confidence of each point of the three-dimensional point cloud based on statistical data of the recognition result of each of the two-dimensional images corresponding to the three-dimensional point cloud.

7. The information processing device according to claim 1, further comprising: A three-dimensional point cloud extraction unit is used to extract a point cloud from the three-dimensional point cloud based on the confidence level of each point of the three-dimensional point cloud.

8. The information processing device according to claim 7, wherein: The three-dimensional point cloud extraction unit extracts the point cloud having a confidence within a predetermined range from the three-dimensional point cloud based on the confidence of each point of the three-dimensional point cloud.

9. The information processing device according to claim 7, wherein: The three-dimensional point cloud extraction unit stores the confidence of each point of the three-dimensional point cloud, and extracts the point cloud from the three-dimensional point cloud based on the stored confidence of each point of the three-dimensional point cloud.

10. The information processing device according to claim 1, further comprising: A three-dimensional point cloud generating unit is used to generate the three-dimensional point cloud.

11. The information processing device according to claim 10, wherein: The three-dimensional point cloud generating unit generates the three-dimensional point cloud according to three-dimensional distance information.

12. The information processing device according to claim 10, wherein: The three-dimensional point cloud generating unit generates the three-dimensional point cloud according to a plurality of two-dimensional images.

13. An information processing method, comprising: performing recognition processing on the three-dimensional point cloud and providing a recognition result for each point of the three-dimensional point cloud; as well as The confidence of the recognition result is calculated for each point of the three-dimensional point cloud and the confidence is given for each point of the three-dimensional point cloud.

14. A method for generating information, comprising: Generate information, the information including: The position information of each point in the 3D point cloud, The recognition result of each point of the three-dimensional point cloud, and The confidence level of the recognition result of each point of the three-dimensional point cloud.

Images