Information processing device, method, and program

Abstract

An information processing device according to one embodiment comprises: a point cloud feature extraction unit that extracts features from each of a plurality of sets of three-dimensional point cloud data; a language feature extraction unit that extracts features from language information; a likelihood calculation unit that calculates a likelihood for each of the plurality of sets of three-dimensional point cloud data on the basis of the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit; and a parameter calculation unit that calculates parameters necessary for aligning each of the plurality of sets of three-dimensional point cloud data, on the basis of the calculation results by the likelihood calculation unit.

Description

Information processing device, method and program

[0001] FIELD Embodiments of the present invention relate to an information processing device, method, and program.

[0002] Efforts are being made to develop point cloud registration techniques with the aim of generating a wider, denser point cloud by aligning the positions and orientations of three-dimensional point cloud data acquired at different locations.

[0003] Point cloud registration techniques can be broadly divided into rule-based methods and machine learning-based methods. Rule-based methods often fall into local solutions when the overlapping area between point clouds is small, making accurate registration difficult. In contrast, machine learning-based methods achieve high registration accuracy by using a machine learning model to estimate corresponding points between point clouds with small overlapping areas (see, for example, Non-Patent Document 1).

[0004] Qin, Zheng, et al. "Geometric transformer for fast and robust point cloud registration." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022.

[0005] While machine learning-based methods have higher alignment accuracy than rule-based methods, they require a large amount of training data, and accuracy drops significantly when unknown data not included in the training data is input.

[0006] This invention was made in light of the above circumstances, and its purpose is to provide an information processing device, method, and program that enables high-precision alignment of three-dimensional point cloud data using machine learning.

[0007] An information processing device according to one aspect of the present invention includes a point cloud feature extraction unit that extracts features for each of a plurality of three-dimensional point cloud data, a language feature extraction unit that extracts features of language information, a likelihood calculation unit that calculates a likelihood for each of the plurality of three-dimensional point cloud data based on the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit, and a parameter calculation unit that calculates parameters required for aligning each of the plurality of three-dimensional point cloud data based on the calculation results by the likelihood calculation unit.

[0008] An information processing method according to one aspect of the present invention is a method performed by an information processing device, and includes: extracting features for each of a plurality of three-dimensional point cloud data using a point cloud feature extraction unit of the information processing device; extracting features of language information using a language feature extraction unit of the information processing device; calculating a likelihood for each of the plurality of three-dimensional point cloud data using a likelihood calculation unit of the information processing device based on the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit; and calculating parameters required for aligning each of the plurality of three-dimensional point cloud data using a parameter calculation unit of the information processing device based on the calculation results by the likelihood calculation unit.

[0009] According to the present invention, it is possible to perform high-precision alignment of three-dimensional point cloud data using machine learning.

[0010] FIG. 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of a procedure of processing operations of an information processing device according to an embodiment of the present invention. FIG. 3 is a diagram showing an application example of a point cloud feature extraction unit. FIG. 4 is a diagram showing an application example of a language feature extraction unit. FIG. 5 is a diagram showing an application example of a likelihood calculation unit. FIG. 6 is a diagram showing an application example of an information processing device according to an embodiment of the present invention when using trained parameters. FIG. 7 is a diagram showing an application example of an information processing device according to an embodiment of the present invention when multiple prompts are input. FIG. 8 is a diagram showing an application example of an information processing device according to an embodiment of the present invention when generating text. FIG. 9 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention.

[0011] An embodiment of the present invention will be described below. In this embodiment, a set of three-dimensional coordinate values ​​that can be acquired by a LiDAR or the like is defined as a "point cloud." In addition, in this embodiment, a natural language, word, or character string is defined as a "prompt."

[0012] 1 is a diagram showing an application example of an information processing device according to an embodiment of the present invention. As shown in FIG. 1, an information processing device 100, which is a point cloud registration processing device according to this embodiment, includes a point cloud feature extraction unit 10, a language feature extraction unit 20, a likelihood calculation unit 30, and a point cloud registration unit 40.

[0013] 2 is a flowchart showing an example of a processing procedure of an information processing device according to an embodiment of the present invention. In this embodiment, first, point cloud feature extraction unit 10 extracts point cloud features, and language feature extraction unit 20 extracts language features (S10). Then, using the extraction results in S10, likelihood calculation unit 30 calculates a likelihood map for the three-dimensional point cloud data (S20). Using this likelihood map, point cloud alignment unit 40 calculates parameters necessary for aligning the point cloud data (S30).

[0014] Further, the details of the functions included in the point cloud feature extraction unit 10, the language feature extraction unit 20, the likelihood calculation unit 30, and the point cloud alignment unit 40 will be listed in order.

[0015] (Point Cloud Feature Extraction Unit 10) The point cloud feature extraction unit 10 outputs point cloud feature quantities, which are characteristics of a point cloud, from an input point cloud, i.e., three-dimensional point cloud data. The point cloud feature extraction unit 10 performs the following input, output, and processing.

[0016] Input: Point cloud A (source of alignment), point cloud B (destination of alignment) Output: Point cloud feature amount A (source of alignment), point cloud feature amount B (destination of alignment) Processing: Point cloud data is input to a point cloud encoder, and feature amounts for each point are extracted

[0017] 3 is a diagram showing an application example of the point cloud feature extraction unit. As shown in FIG. 3, the point cloud feature extraction unit 10 has two point cloud encoders capable of performing the same processing, namely, a first point cloud encoder 10-1 and a second point cloud encoder 10-2. The first point cloud encoder 10-1 of the point cloud feature extraction unit 10 inputs point cloud A, extracts point cloud feature amount A, which is a feature amount of point cloud A, and outputs this. The second point cloud encoder 10-2 of the point cloud feature extraction unit 10 inputs point cloud B, extracts point cloud feature amount B, which is a feature amount of point cloud B, and outputs this.

[0018] The feature quantities of a point cloud can be obtained using existing methods, such as the methods disclosed in the following references 1, 2, or 3. The feature quantities that can be extracted by a point cloud encoder are geometric features obtained from a point cloud and information on nearby points expressed as vectors or the like. The point cloud feature quantities described above are also called point cloud feature vectors.

[0019] (Reference 1) Wang, Yue, et al. "Dynamic graph cnn for learning on point clouds.", ACM Transactions on Graphics, no. 38, issue. 5, pp.1-12, 2019. (Reference 2) Thomas, Hugues, et al. "Kpconv: Flexible and deformable convolution for point clouds.", Proceedings of the IEEE / CVF international conference on computer vision, 2019. (Reference 3) Choy, Christopher, et al. "4d spatio-temporal convnets: Minkowski convolutional neural networks.", Proceedings of the IEEE / CVF international conference on computer vision and pattern recognition, 2019.

[0020] (Language Feature Extraction Unit 20) The language feature extraction unit 20 outputs language features, which are characteristics of linguistic information, from the input prompt. The language feature extraction unit 20 performs the following input, output, and processing.

[0021] Input: Prompt Output: Language features Processing: Input the prompt to the language encoder and extract language features

[0022] Fig. 4 is a diagram showing an application example of the language feature extraction unit. As shown in Fig. 4, the language feature extraction unit 20 has a language encoder 20-1. The language encoder 20-1 of the point cloud feature extraction unit 10 inputs a prompt, extracts language features that are features of the prompt, and outputs them.

[0023] Language features can be obtained using existing methods, such as those disclosed in the following references 4 and 5. Features that can be extracted by a language encoder are representations of semantic features, contextual features, and character string features of a language as vectors or the like. The above-mentioned language features are also called language feature vectors.

[0024] (Reference 4) Radford, Alec, et al. "Improving language understanding by generative pre-training.", 2018. (Reference 5) Vaswani, Ashish, et al. "Attention is all you need.", Advances in neural information processing systems, 2017.

[0025] (Likelihood Calculation Unit 30) The likelihood calculation unit 30 calculates the likelihood of a point cloud from the similarity between the point cloud feature obtained by the point cloud feature extraction unit 10 and the language feature obtained by the language feature extraction unit 20. The likelihood calculation unit 30 performs the following input, output, and processing.

[0026] Input: Point cloud feature A (source of alignment), point cloud feature B (destination of alignment), language feature Output: Likelihood map A (source of alignment), likelihood map B (destination of alignment) Processing: Calculate similarity etc. from point cloud feature and language feature, and output as likelihood for each point cloud

[0027] 5 is a diagram illustrating an application example of the likelihood calculation unit 30. As shown in FIG. 5, the likelihood calculation unit 30 has two “language-point cloud likelihood calculation units” that can perform the same processing, namely, a first language-point cloud likelihood calculation unit 30-1 and a second language-point cloud likelihood calculation unit 30-2.

[0028] The first language-point cloud likelihood calculation unit 30-1 of the likelihood calculation unit 30 inputs the point cloud feature A from the point cloud feature extraction unit 10 and the language feature from the language feature extraction unit 20, calculates a likelihood map A, and outputs it. The second language-point cloud likelihood calculation unit 30-2 of the likelihood calculation unit 30 inputs the point cloud feature B from the point cloud feature extraction unit 10 and the language feature from the language feature extraction unit 20, calculates a likelihood map B, and outputs it.

[0029] The likelihood map can be calculated using likelihoods obtained from existing methods, such as the cosine similarity method disclosed in the following references 6 and 7. The likelihood map can also be calculated using likelihoods obtained from existing methods, such as the neural network method disclosed in the above-mentioned reference 5.

[0030] (Reference 6) Radford, Alec, et al. "Learning transferable visual models from natural language supervision.", International conference on machine learning, 2021. (Reference 7) Peng, Songyou, et al. "Openscene: 3d scene understanding with open vocabulary.", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023.

[0031] (Point Cloud Alignment Unit 40) The point cloud alignment unit 40 performs the following input, output, and processing: Input: point cloud A, point cloud B, likelihood map A (alignment source), likelihood map B (alignment destination) Output: rotation and translation parameters Processing: search for corresponding points between point clouds from the distribution of the input likelihood map, calculate rotation and translation parameters required for aligning the point clouds, and output Note that alignment can be performed using an existing method, for example, the disclosures in References 8 or 9 below.

[0032] (Reference 8) Besl, Paul J., et al. "Method for registration of 3-D shapes." Sensor fusion IV: control paradigms and data structures. Vol. 1611. Spie, 1992. (Reference 9) MA Fischler and RC Bolles, "Random sample consensus: aparadigm for model fitting with applications to image analysis and automated cartography," Communications of the ACM, vol. 24, no. 6, pp. 381‐395, 1981.

[0033] In this embodiment, an encoder that requires training can be used in the point cloud feature extraction unit 10 and the language feature extraction unit 20. When using this encoder that requires training, the encoder can use existing methods, such as pre-trained parameters disclosed in the above-mentioned Reference 7, without undergoing training for aligning point clouds.

[0034] Fig. 6 is a diagram showing an application example when trained parameters are used in an information processing device according to an embodiment of the present invention. In the example shown in Fig. 6, a trained parameter DB (database) 15 is connected to a point cloud feature extraction unit 10, and a trained parameter DB 25 is connected to a language feature extraction unit 20. The point cloud feature extraction unit 10 extracts point cloud features using pre-trained parameters stored in the trained parameter DB 15. The language feature extraction unit 20 extracts language features using pre-trained parameters stored in the trained parameter DB 25.

[0035] As described above, by using pre-trained parameters capable of outputting general-purpose features in the point cloud feature extraction unit 10 and the language feature extraction unit 20 when inferring features, the likelihood increases for point cloud features that co-occur with language features. By utilizing this property, points with high likelihoods obtained from two point cloud features for one language feature can become corresponding points when aligning point clouds. Therefore, even if a point cloud that has not been trained by the point cloud feature extraction unit 10 is input, alignment is possible by using the likelihood obtained from the language features.

[0036] Next, an example of a function extension in this embodiment will be described. (Function Extension (1)) Generating and Aligning Multiple Likelihood Maps from Multiple Prompts Figure 7 is a diagram showing an application example of an information processing device according to an embodiment of the present invention when multiple prompts are input. In the example shown in Figure 7, unlike the configuration shown in Figure 1, the language feature extraction unit 20 inputs multiple random prompts, extracts language features for each of these input prompts, and outputs them as multiple language features to the likelihood calculation unit 30.

[0037] The likelihood calculation unit 30 receives the point cloud feature A and the above-described plurality of language features, calculates a likelihood map for each of the plurality of language features, and outputs the likelihood maps A.

[0038] Furthermore, the likelihood calculation unit 30 inputs the point cloud feature B and the above-mentioned multiple language features, calculates a likelihood map obtained for each of the multiple language features, and outputs the likelihood maps B.

[0039] More specifically, when the plurality of prompts are two prompts, a first language feature and a second language feature are output from the language feature extraction unit 20. Then, based on the point cloud feature A and the first language feature, the likelihood calculation unit 30 outputs a first likelihood map A, and based on the point cloud feature A and the second language feature, the likelihood calculation unit 30 outputs a second likelihood map B. The same applies to the likelihood map B.

[0040] In this way, multiple random prompts can be input to calculate multiple likelihoods, which can prevent the system from falling into a local optimum.

[0041] (Function Extension (2)) Generating Multiple Texts from a Prompt FIG. 8 is a diagram showing an example of application of an information processing device according to an embodiment of the present invention when generating text. In the example shown in FIG. 8, unlike the configuration shown in FIG. 1, a text generation unit 50 is provided before the language feature extraction unit 20. The text generation unit 50 inputs a prompt and generates multiple random texts related to the prompt from the prompt.

[0042] The text generation unit 50 performs the following input, output, and processing: Input: Prompt Output: Multiple texts Processing: Generate multiple texts according to the input prompts

[0043] A trained large-scale language model can be used to generate text. For example, the text generation unit 50 inputs domain information of point clouds into the model and outputs multiple random texts related to this information. Also, a prompt to output random texts is input into the model, and the model outputs multiple random texts. The language feature extraction unit 20 extracts language features for each of the multiple random texts and outputs them to the likelihood calculation unit 30 as multiple language features. The subsequent processing is similar to the processing in the example shown in FIG. 7.

[0044] In the example shown in Fig. 8, it is possible to calculate multiple likelihoods based on multiple random texts. Using these multiple likelihoods makes it possible to prevent the system from falling into a local optimum.

[0045] In the embodiment described above, even if data is not included in the learning data, it is possible to suppress a decrease in the accuracy of machine learning-based point cloud alignment by inputting an appropriate prompt.

[0046] 9 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention. In the example shown in FIG. 9, the information processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0047] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0048] An input device 500 and an output device 600 attached to the information processing device 100 and used by a user or the like are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 500 such as a keyboard, touch panel, touchpad, mouse, or the like, and can output output data to an output device 600 including a display device using a liquid crystal or organic electroluminescence (EL) display, or the like, for display. The input device 500 and the output device 600 may be devices built into the information processing device 100, or may be input devices and output devices of other information terminals that can communicate with the information processing device 100 via a network.

[0049] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and can store programs necessary to execute various control processes, etc., according to one embodiment.

[0050] The data memory 112 is a tangible storage medium that is, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and can be used to store various data or information acquired and created during various processing steps.

[0051] An information processing apparatus 100 according to one embodiment of the present invention can be configured as a data processing apparatus having the units shown in FIG. 1 as software-based processing function units.

[0052] The information storage unit used as a work memory or the like by each unit of the information processing device 100 can be configured by using the data memory 112 shown in Fig. 9. However, these configured storage areas are not essential components within the information processing device 100, and may be areas provided in, for example, an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0053] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0054] The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), or semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables or data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by controlling the operation of this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0055] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0056] REFERENCE SIGNS LIST 100... Information processing device 10... Point cloud feature extraction unit 15, 25... Learned parameter DB 20... Language feature extraction unit 30... Likelihood calculation unit 40... Point cloud alignment unit 50... Text generation unit

Claims

1. An information processing device comprising: a point cloud feature extraction unit that extracts features for each of a plurality of three-dimensional point cloud data; a language feature extraction unit that extracts features of language information; a likelihood calculation unit that calculates a likelihood for each of the plurality of three-dimensional point cloud data based on the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit; and a parameter calculation unit that calculates parameters necessary for aligning each of the plurality of three-dimensional point cloud data based on the calculation results by the likelihood calculation unit.

2. The information processing device according to claim 1, wherein the point cloud feature extraction unit uses trained parameters to extract features for each of the plurality of three-dimensional point cloud data, and the language feature extraction unit uses trained parameters to extract features of the language information.

3. The information processing device described in claim 1, wherein the language feature extraction unit extracts features of each of the multiple pieces of language information, and the likelihood calculation unit calculates a number of likelihoods corresponding to the number of the multiple pieces of language information for each of the multiple three-dimensional point cloud data based on the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit.

4. An information processing device as described in claim 1, further comprising a text generation unit that generates a plurality of texts related to the linguistic information, wherein the language feature extraction unit extracts features of each of the plurality of texts generated by the text generation unit, and the likelihood calculation unit calculates a likelihood of a number corresponding to the number of the plurality of texts for each of the plurality of three-dimensional point cloud data based on the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit.

5. A method performed by an information processing device, comprising: extracting features for each of a plurality of three-dimensional point cloud data using a point cloud feature extraction unit of the information processing device; extracting features of language information using a language feature extraction unit of the information processing device; calculating a likelihood for each of the plurality of three-dimensional point cloud data using a likelihood calculation unit of the information processing device based on the extraction results by the point cloud feature extraction unit and the extraction results by the language feature extraction unit; and calculating parameters required for aligning each of the plurality of three-dimensional point cloud data using a parameter calculation unit of the information processing device based on the calculation results by the likelihood calculation unit.

6. An information processing program that causes a processor to function as each part of the information processing device according to any one of claims 1 to 4.

Images