Information processing device, information processing method, and information processing program

Abstract

An information processing device according to an embodiment of the present invention includes an acquisition unit, a recognition unit, a specification unit, and a removal unit. The acquisition unit acquires point cloud data to be processed. The recognition unit executes, by using a trained model trained to estimate a class label of a point cloud, processing of allocating the class label to the point cloud data. The specification unit specifies, from among the point cloud data, a noise point cloud which is a point cloud to which the class label is not allocated. The removal unit removes the noise point cloud.

Description

Information processing device, information processing method, and information processing program

[0001] The embodiments relate to an information processing device, an information processing method, and an information processing program.

[0002] There is a 3D point cloud data technology that uses LiDAR (Light Detection and Ranging) to measure objects and people, generate a 3D space represented by points, and then color each point using video from a camera to represent the 3D space and the objects and people within the space as a collection of points. Because 3D point cloud data is 3D information at the time of measurement, it is possible to construct a 3D space at a lower cost than constructing a 3D space from a photograph (see, for example, Non-Patent Document 1).

[0003] Nippon Telegraph and Telephone Corporation et al., "Launch of the 'TENGUN Ogijima Project' to Promote Regional Co-creation - Joint Study Begins Aiming to Create and Expand a Connected Population through a Photorealistic 'Ogijima' Metaverse Realized by IOWN," [online], November 15, 2022, [Retrieved May 14, 2024], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 11 / 15 / 221115b.html>

[0004] However, 3D point cloud data often involves capturing images using a camera while irradiating radar in various directions, resulting in a huge amount of data. Meanwhile, depending on the measurement conditions, dust particles in the air or unexpected reflections may cause point cloud data noise in areas where no objects exist. Manually removing such noisy point cloud data is extremely costly.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an information processing device, an information processing method, and an information processing program that can realize efficient data processing.

[0006] An information processing apparatus according to an embodiment includes an acquisition unit, a recognition unit, an identification unit, and a removal unit. The acquisition unit acquires point cloud data to be processed. The recognition unit assigns class labels to the point cloud data using a trained model that has been trained to estimate class labels of the point cloud. The identification unit identifies noise point clouds, which are point clouds to which the class labels have not been assigned, from the point cloud data. The removal unit removes the noise point clouds.

[0007] According to the embodiments, it is possible to provide an information processing device, an information processing method, and an information processing program that can realize efficient data processing.

[0008] FIG. 1 is a block diagram showing an information processing apparatus according to a first embodiment. FIG. 2 is a flowchart showing an example of operation of the information processing apparatus according to the first embodiment. FIG. 3 is a diagram showing a specific example of noise removal processing by the information processing apparatus according to the first embodiment. FIG. 4 is a block diagram showing an information processing apparatus according to a second embodiment. FIG. 5 is a flowchart showing an example of operation of the information processing apparatus according to the second embodiment. FIG. 6 is a diagram showing a specific example of noise removal processing by the information processing apparatus according to the second embodiment. FIG. 7 is a block diagram showing the hardware configuration of the information processing apparatus according to this embodiment.

[0009] Each embodiment will be described below with reference to the drawings. Each embodiment illustrates an apparatus or method for embodying the technical idea of ​​the invention. The drawings are schematic or conceptual. In the following, the same reference numerals are used to designate components having substantially the same functions and configurations. The numbers following the letters that make up the reference numerals are used to distinguish between elements that are referred to by the reference numerals containing the same letters and that have similar configurations. When there is no need to distinguish between elements indicated by reference numerals containing the same letters or numbers, these elements will be referred to by reference numerals containing only letters or numbers.

[0010] First Embodiment An information processing device 1 according to a first embodiment will be described with reference to the block diagram of Fig. 1. The information processing device 1 according to the first embodiment includes an acquisition unit 11, a recognition unit 12, an identification unit 13, a removal unit 14, an output unit 15, and a storage unit 16.

[0011] The acquisition unit 11 acquires point cloud data to be processed. The point cloud data includes three-dimensional position information (x, y, z) about the surfaces of objects and people. Additionally, additional information of the point cloud data may include color information captured by a camera when measuring the point cloud data.

[0012] The recognition unit 12 assigns class labels to the target point cloud data using a trained model that has been trained to estimate class labels for each point cloud from the point cloud data. The identification unit 13 identifies noise point clouds, which are point clouds to which no class label has been assigned, from the target point cloud data.

[0013] The removal unit 14 removes noise point clouds from the point cloud data to be processed and generates processed point cloud data. The output unit 15 outputs the processed point cloud data to an external device. The storage unit 16 stores the trained model, the point cloud data, the processed point cloud data, etc.

[0014] Next, a processing example of the information processing apparatus 1 according to the first embodiment will be described with reference to the flowchart of FIG.

[0015] In step SA1, the acquisition unit 11 acquires point cloud data to be processed. For example, point cloud data measured by LiDAR may be acquired.

[0016] In step SA2, the recognition unit 12 uses the trained model to assign class labels to the point cloud data to be processed. Class labels are labels that classify the object each point in the point cloud data represents, such as road, building, tree, or person. The class label assignment process is a so-called semantic segmentation process, in which a class label is assigned to each point in the point cloud data to be processed. The trained model may be any model that has been trained to input point cloud data and perform a classification process that assigns a class label to each point in the point cloud data. For example, the method and trained model disclosed in Japanese Patent No. 7424509 may be used. Here, class labels are not assigned to point clouds that cannot be classified using class labels using the trained model.

[0017] In step SA3, the identification unit 13 identifies noise point clouds from the point cloud data to which class labels have been assigned. Specifically, point clouds to which no class labels have been assigned are identified as noise point clouds. Point clouds to which no class labels have been assigned are, for example, point clouds that exist in areas where there is nothing, or point clouds that exist at distances (depths) that are clearly different from the class labels assigned to surrounding point clouds.

[0018] In step SA4, the removal unit 14 removes (deletes) the identified noise point cloud from the point cloud data to be processed, generating processed point cloud data. In step SA5, the output unit 15 outputs the processed point cloud data to an external device. The storage unit 16 may store the processed point cloud data.

[0019] Next, a specific example of the noise removal process of the information processing device 1 according to the first embodiment will be described with reference to the conceptual diagram of FIG.

[0020] 3 shows an example of 3D point cloud data to be processed, which is 3D point cloud data 31 obtained by measuring a car traveling on a roadway. For the sake of convenience, the outline of the object is shown with a solid line, but it is assumed that the object is actually measured as point cloud data.

[0021] The central diagram in Figure 3 shows segmented point cloud data 32 after the recognition unit 12 has assigned a class label to each point in the 3D point cloud data. In the segmented point cloud data 32, areas classified with the same class label, such as automobiles, walls, trees, and roads, are shown with the same type of hatching. Here, point clouds to which no class label has been assigned exist at the top of the segmented point cloud data 32. The identification unit 13 identifies such point clouds to which no class label has been assigned as noise point clouds 33.

[0022] 3 shows processed point cloud data 34 from which the noise point cloud 33 has been removed by the removal unit 14. In this way, by using the trained model, high-quality point cloud data that does not include the noise point cloud 33 can be generated.

[0023] According to the first embodiment described above, the recognition unit assigns a class label to each point in the point cloud data to be processed. The identification unit identifies point clouds to which no class label has been assigned from the point cloud data to be processed as noise point clouds. The removal unit removes the noise point clouds. By being able to identify noise point clouds using a trained model, it is possible to achieve high-quality point cloud data and segmentation processing while significantly reducing work costs, rather than the conventional manual work. In other words, it is possible to achieve efficient data processing.

[0024] (Second embodiment) In the first embodiment, it is assumed that point clouds to which no class label is assigned are uniformly deleted as noise point clouds using a trained model. However, in the second embodiment, a user assigns a flag to a noise point cloud identified using a trained model in a three-dimensional data space, and the flagged noise point cloud is removed.

[0025] An information processing device 1 according to the second embodiment will be described with reference to the block diagram of Fig. 4. The information processing device 1 according to the second embodiment includes an acquisition unit 11, a recognition unit 12, an identification unit 13, an arrangement unit 21, an assignment unit 22, a removal unit 14, an output unit 15, and a storage unit 16.

[0026] The acquisition unit 11 acquires point cloud data to be processed. The recognition unit 12 assigns class labels to the point cloud data to be processed using a trained model that has been trained to estimate class labels of point clouds from the point cloud data. The identification unit 13 identifies noise point clouds, which are point clouds to which no class labels have been assigned, from the point cloud data.

[0027] The arrangement unit 21 arranges the point cloud data, to which the class label assignment process has been performed, in a three-dimensional virtual space. The assignment unit 22 assigns a flag to the noise point cloud in response to a user instruction, based on viewpoint information that can be moved in the three-dimensional virtual space.

[0028] The removal unit 14 removes flagged noise point clouds to generate processed point cloud data. The output unit 15 outputs the processed point cloud data to an external device. The storage unit 16 stores the trained model, point cloud data, flagged noise point clouds, and processed point cloud data.

[0029] Next, a specific example of the noise removal process of the information processing device 1 according to the second embodiment will be described with reference to the flowchart of Fig. 5. The processes of steps SA1 to SA3 and step SA5 are the same as those in the first embodiment.

[0030] In step SB1, the arrangement unit 21 arranges the point cloud data, to which class labels have been assigned, in a three-dimensional virtual space. Specifically, the point cloud data is mapped into a three-dimensional virtual space, and a data space is assumed in which a user can freely move around the three-dimensional virtual space from a first-person perspective. In step SB2, the assignment unit 22 assigns flags to noise point clouds specified in the three-dimensional virtual space based on user instructions using a mouse, stylus pen, or the like. In step SB3, the removal unit 14 removes (deletes) the flagged noise point clouds from the point cloud data to be processed, thereby generating processed point cloud data.

[0031] Next, a specific example of the noise removal process of the information processing device 1 according to the second embodiment will be described with reference to the conceptual diagram of FIG.

[0032] The upper diagram of Figure 6 shows three-dimensional point cloud data 60, an example of three-dimensional point cloud data arranged in a three-dimensional virtual space. The three-dimensional point cloud data 60 represents a stereoscopic image currently displayed on a display in a three-dimensional virtual space in which a user can freely move from a 360-degree viewpoint. For ease of explanation, the contours of objects are shown with solid lines, but it is assumed that they are actually measured as point cloud data. In the three-dimensional point cloud data 60, a user's viewpoint 62 is indicated by an arrow. Furthermore, the viewpoint positions of other users are represented in the three-dimensional virtual space as avatars 63. In other words, the direction in which the avatar 63 is facing represents the viewpoint direction of the other users, and the movement of the avatar 63 represents the movement of the other users' viewpoints. Here, in the upper diagram of Figure 6, it is assumed that noise point clouds 61 exist in multiple locations.

[0033] The central diagram of FIG. 6 shows an example in which a user assigns a flag 64 to a noise point cloud 61 in three-dimensional point cloud data 60 by surrounding the noise point cloud 61 with a mouse, or with the user's finger or stylus pen in the case of a tablet PC or smartphone. It is assumed that the flag 64 is assigned to the noise point cloud 61 by other users (avatars 63) who exist in the same three-dimensional virtual space while moving together. Furthermore, the flags 64 assigned to each user may be assigned in a distinguishable manner, such as by changing the color of the flag 64. Note that the flag 64 is not limited to circumscribing the noise point cloud 61. Any label assignment method, such as filling in the area of ​​the noise point cloud 61, may be used as long as it is possible to confirm that the user and other users have assigned annotations to the noise point cloud 61.

[0034] In the lower diagram of Fig. 6, the removal unit 14 removes the noise point cloud 61 to which the user has assigned a flag 64, thereby generating processed point cloud data. Note that the targets to which the user may assign the flag 64 are not limited to noise point clouds. The user may also assign the flag 64 to point clouds to which the class label classified by the trained model is incorrect, point clouds to which a point cloud identified as a noise point cloud should be assigned a different class label, or point clouds that the user can visually confirm. Furthermore, the storage unit 16 may store the point clouds to which the flag 64 has been assigned, and the point clouds may be fed back as learning data when training a machine learning model that classifies class labels.

[0035] According to the second embodiment described above, multiple users can walk through, in other words, move and explore in a three-dimensional virtual space, while closely examining the noise point cloud identification status, and higher-quality processed point cloud data can be generated. Although the second embodiment requires manual work, the noise point cloud can be roughly identified in advance using a trained model, limiting the areas where users need to stop processing and significantly reducing work costs. In other words, efficient data processing can be achieved.

[0036] Next, an example of the hardware configuration of the information processing device 1 according to this embodiment will be described with reference to the block diagram shown in FIG.

[0037] As shown in FIG. 11, the information processing device 1 includes, for example, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a display 104, a communication interface 105, and storage 106.

[0038] The CPU 101 is an integrated circuit capable of executing various programs and controls the overall operation of the information processing device 1. The ROM 102 is, for example, a non-volatile semiconductor memory and stores programs and control data for controlling the information processing device 1. The RAM 103 is, for example, a volatile semiconductor memory and is used as a work area for the CPU 101. The display 104 is, for example, a liquid crystal display or a touch panel display and displays information. The communication interface 105 is, for example, an input / output component for exchanging information with the outside, such as a USB, an HDMI (registered trademark), or a network interface. The storage 106 is a non-volatile storage device. The storage 106 stores system software and the like for the information processing device 1. The CPU 101 may also be called a "processor."

[0039] In the above embodiments, the CPU 101 of the information processing device 1 may be another circuit (or processor). For example, the information processing device 1 may include a GPU (Graphics Processing Unit), an NPU (Neural Network Processing Unit), an MPU (Micro Processing Unit), or the like instead of a CPU. Each of the processes described in each embodiment may be realized by dedicated hardware. The processes of the information processing device 1 may be a mixture of processes executed by software and processes executed by hardware, or may be only one of them.

[0040] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0041] DESCRIPTION OF SYMBOLS 1... Information processing device 11... Acquisition unit 12... Recognition unit 13... Identification unit 14... Removal unit 15... Output unit 16... Storage unit 21... Placement unit 22... Assignment unit 31, 60... Three-dimensional point cloud data 32... Segmented point cloud data 33, 61... Noise point cloud 34... Processed point cloud data 62... User viewpoint 63... Avatar 64... Flag 101... CPU 102... ROM 103... RAM 104... Display 105... Communication interface 106... Storage

Claims

1. An information processing device comprising: an acquisition unit that acquires point cloud data to be processed; a recognition unit that executes a process of assigning class labels to the point cloud data using a learned model trained to estimate the class labels of the point cloud; an identification unit that identifies noise point clouds, which are point clouds from the point cloud data to which the class labels have not been assigned; and a removal unit that removes the noise point clouds.

2. An information processing device comprising: an acquisition unit that acquires point cloud data to be processed; a recognition unit that performs a process of assigning class labels to the point cloud data using a learned model trained to estimate the class labels of the point cloud; an identification unit that identifies noise point clouds, which are point clouds from the point cloud data that have not been assigned a class label; an arrangement unit that arranges the point cloud data to which the class label assignment process has been performed in a three-dimensional virtual space; an assignment unit that assigns flags to the noise point clouds in response to user instructions based on viewpoint information that can be moved in the three-dimensional virtual space; and a removal unit that removes the noise point clouds to which the flags have been assigned.

3. An information processing method, comprising: an acquisition means for acquiring point cloud data to be processed; a classification means for assigning class labels to the point cloud data using a learned model trained to estimate class labels of point clouds; an identification means for identifying noise point clouds, which are point clouds to which no class labels have been assigned, from the point cloud data; and a removal means for removing the noise point clouds.

4. An information processing program for causing a computer to function as each part of the information processing device according to claim 1.

Images