Information processing system, information processing device, terminal, and information processing method

Abstract

This invention provides an information processing system, information processing device, terminal, and information processing method that can efficiently utilize large-capacity point cloud data. [Solution] The system performs a compressed point cloud information generation process, which involves encoding the coordinates of the vertices of point cloud data measured in three-dimensional space to irreversibly compress the point cloud data and generate compressed point cloud information represented as a hierarchical structure, and a rendering process that searches for point cloud data in three-dimensional space that the user requests from the compressed point cloud information generated by the compressed point cloud information generation process, and then renders image information based on the retrieved point cloud data to the user's terminal.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing device, a terminal, and an information processing method, and particularly to an information processing system, an information processing device, a terminal, and an information processing method for processing point cloud data.

Background Art

[0002] For example, there is known a measurement technique in which a laser beam is irradiated onto an object, and based on information on the reflected light of the irradiated laser beam, the distance to the object in three-dimensional space, its shape, etc. are grasped to obtain point cloud data.

[0003] Each point constituting this point cloud data has position information (such as x, y, z coordinates, etc.) of the object in three-dimensional space, and is measured densely enough to grasp the surface shape of the object, thus constituting point cloud data. As a result, the data volume becomes very large, and it may be difficult to handle the point cloud data.

[0004] As a countermeasure in such a case, Patent Document 1 proposes a technique for generating compressed three-dimensional point cloud data representing the point cloud data obtained based on measurement in a hierarchical structure.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, when processing large-capacity point cloud data, while compressing the point cloud data, it is necessary to expand the compressed point cloud data. Also, when expanding large-capacity compressed point cloud data, a load corresponding to the capacity (such as an increase in the time for loading the point cloud data, etc.) will occur, and there is concern that efficient processing of the point cloud data will be hindered.

[0007] This invention has been made in view of the above circumstances, and aims to provide an information processing system, information processing device, terminal, and information processing method that can efficiently utilize large amounts of point cloud data. [Means for solving the problem]

[0008] To achieve the above objective, the information processing system according to the present invention performs a compressed point cloud information generation process that encodes the coordinates of the vertices of point cloud data measured in three-dimensional space, irreversibly compresses the point cloud data, and generates compressed point cloud information represented as a hierarchical structure; and a rendering process that searches for point cloud data in three-dimensional space requested by the user from the compressed point cloud information generated in the compressed point cloud information generation process, and renders image information based on the retrieved point cloud data on the user's terminal.

[0009] According to this method, by compressing point cloud data measured in three-dimensional space and generating compressed point cloud information, even large-capacity point cloud data can be easily handled, such as for storage or transfer.

[0010] Furthermore, by representing the compressed point cloud data as a hierarchical structure, the point cloud data in three-dimensional space can be clearly distinguished, making it easy for users to search for the point cloud data they require.

[0011] Therefore, large amounts of point cloud data can be used efficiently.

[0012] The rendering process performed by this information processing system renders image information based on point cloud data covering a range that matches the user's field of view in three-dimensional space onto the user's terminal.

[0013] The compressed point cloud information processed by this information processing system consists of hierarchical management information, metadata, node data relating to the nodes of the hierarchical structure, and compressed point cloud data. The rendering process refers to the metadata and node data based on the management information, and searches for point cloud data based on the referenced node data.

[0014] Furthermore, the drawing process may involve searching for point cloud data based on the starting offset and size.

[0015] The compressed point cloud information generation process performed by this information processing system compresses point cloud data and generates compressed point cloud information that represents the octvine structure as a hierarchical structure.

[0016] The drawing process performed by this information processing system involves arbitrarily adjusting the dimensions of the points that make up the point cloud data, and then drawing image information based on the point cloud data with adjusted point dimensions to the user's terminal.

[0017] To achieve the above objective, the information processing device according to the present invention performs a compressed point cloud information generation process that encodes the coordinates of the vertices of point cloud data measured in three-dimensional space, irreversibly compresses the point cloud data, and generates compressed point cloud information represented as a hierarchical structure. The compressed point cloud information generated by the compressed point cloud information generation process comprises management information for the hierarchical structure, metadata, node data relating to the nodes of the hierarchical structure, and compressed point cloud data.

[0018] To achieve the above objective, the terminal according to the present invention performs a drawing process that searches for point cloud data in three-dimensional space requested by the user from compressed point cloud information, which is obtained by encoding the coordinates of the vertices of point cloud data measured in three-dimensional space, irreversibly compressing the point cloud data, and representing it as a hierarchical structure, and then draws image information based on the retrieved point cloud data. The drawing process draws image information based on point cloud data within a range that matches the user's field of view in three-dimensional space.

[0019] The information processing method according to the present invention for achieving the above object is such that an information processing apparatus implemented by a computer encodes the coordinates of vertices of point cloud data measured in a three-dimensional space, irreversibly compresses the point cloud data, and generates compressed point cloud information represented as a hierarchical structure by executing a compressed point cloud information generation process. A terminal implemented by a computer different from the above computer executes a drawing process of retrieving point cloud data in a three-dimensional space desired by a user from the compressed point cloud information generated in the compressed point cloud information generation process and drawing image information based on the retrieved point cloud data on the user's terminal.

Effect of the Invention

[0020] According to this invention, a large amount of point cloud data can be efficiently utilized.

Brief Explanation of Drawings

[0021] [Figure 1] It is a block diagram for explaining an outline of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] Similarly, it is a block diagram for explaining an outline of the configuration of a computer implementing an information processing apparatus and a user terminal of the information processing system according to this embodiment. [Figure 3] Similarly, it is a block diagram for explaining an outline of the functions of an information processing apparatus of the information processing system according to this embodiment. [Figure 4] Similarly, it is a diagram for explaining an outline of the processing of an information processing apparatus of the information processing system according to this embodiment. [Figure 5] Similarly, it is a diagram for explaining an outline of the compressed point cloud information processed by the information processing system according to this embodiment. [Figure 6] Similarly, it is a diagram for explaining an outline of the compressed point cloud information processed by the information processing system according to this embodiment. [Figure 7] Similarly, it is a diagram for explaining an outline of the functions of a user terminal of the information processing system according to this embodiment. [Figure 8]Similarly, this is a flowchart illustrating the general processing steps of the information processing system according to this embodiment. [Figure 9] Similarly, this is a flowchart illustrating the outline of the drawing process of the information processing system according to this embodiment. [Modes for carrying out the invention]

[0022] Next, an information processing system according to an embodiment of the present invention will be described based on Figures 1 to 9.

[0023] Figure 1 is a block diagram illustrating the general configuration of the information processing system according to this embodiment. As shown in the figure, the information processing system 10 mainly consists of an information processing device 20 and a plurality of user terminals 30, which are connected to each other via a network N such as the Internet.

[0024] In this embodiment, the information processing device 20 is managed by a service provider 1 that provides services using the information processing system 10, and the user terminals 30 are owned by multiple users 2 who use the services provided by the service provider 1 that utilize the information processing system 10.

[0025] In this embodiment, the services using the information processing system 10 are envisioned to be, for example, a service that provides an arbitrary image to the user terminal 30 and displays it on the user terminal 30, or a service that provides the user terminal 30 with a virtual space created by overlaying virtual information objects onto an image of the real world, but are not limited to these.

[0026] On the other hand, in this embodiment, User 2 is a business operator that needs to display point cloud data on the user terminal 30 using a service provided by Business Operator 1 on the user terminal 30. This includes, for example, businesses that develop and provide services such as online games or online tours using virtual spaces that are run on player terminals owned by players (not shown) to the player terminals.

[0027] Next, we will describe the specific configuration of each part of the information processing system 10.

[0028] In this embodiment, the information processing device 20 and the user terminal 30 are implemented by a computer having substantially the same hardware configuration, such as a desktop or notebook computer.

[0029] Figure 2 is a block diagram illustrating the general configuration of a computer. As shown in the figure, the computer mainly consists of a processor 101, memory 102, storage 103, a transceiver 104, and an input / output unit 105, which are electrically connected to each other via a bus 106.

[0030] The processor 101 is a computing unit that controls the operation of the computer, controls the transmission and reception of data between each element, and performs processing necessary for the execution of application software and programs.

[0031] In this embodiment, the processor 101 is, for example, a CPU (Central Processing Unit), which executes application software and programs deployed in the memory 102 described below to perform various processes.

[0032] Memory 102 is implemented by a main memory system composed of volatile memory devices such as DRAM (Dynamic Random Access Memory).

[0033] This memory 102 is used as a workspace for the processor 101, while also storing the BIOS (Basic Input / Output System) that runs when the computer starts up, as well as various configuration information.

[0034] Storage 103 stores data used for various processes by application software and programs.

[0035] The transmitting / receiving unit 104 connects the computer to the network N. This transmitting / receiving unit 104 may support wireless communication standards such as Wi-Fi, or it may be equipped with a short-range communication interface such as Bluetooth® or BLE (Bluetooth Low Energy).

[0036] The input / output unit 105 is connected to information input devices such as keyboards and mice, and output devices such as displays, as needed. In this embodiment, a keyboard, mouse, and display are connected, respectively.

[0037] Furthermore, in this embodiment, the computer implementing the user terminal 30 may also have a physical camera connected to the input / output unit 105 that can acquire any image (including both moving and still images).

[0038] The bus 106 transmits, for example, address signals, data signals, and various control signals between the connected processor 101, memory 102, storage 103, transceiver 104, and input / output unit 105.

[0039] In this embodiment, the computer on which the information processing device 20 is implemented may be one that is logically realized in a cloud environment.

[0040] Figure 3 is a block diagram illustrating the general functions of the information processing device 20 of the information processing system 10. As shown in the figure, the information processing device 20 comprises an input receiving unit 21, a compressed point cloud information generation unit 22, a compressed point cloud information storage unit 23, and an output unit 24.

[0041] The input receiving unit 21, the compressed point cloud information generation unit 22, and the output unit 24 are realized by executing a program stored in the memory 102 with the processor 101, and the compressed point cloud information storage unit 23 is realized by partitioning the storage area of ​​the storage 103.

[0042] In this embodiment, the input receiving unit 21 receives point cloud data of any object measured in three-dimensional space. This point cloud data is data that has been preprocessed, such as noise removal, from primary data acquired by, for example, a ground-based three-dimensional laser scanner (TLS) installed on the ground, an aerial laser scanner mounted on an aircraft such as a drone, or a three-dimensional scanner mounted on a portable information terminal (smartphone), and contains coordinate information and color information.

[0043] In this embodiment, the compressed point cloud information generation unit 22 performs a process to generate compressed point cloud information that includes, as part of the data structure, the point cloud data obtained by encoding the coordinates of the vertices of the point cloud data received by the input receiving unit 21 and irreversibly compressing it (compressed point cloud information generation process).

[0044] Figure 4 is a diagram illustrating the general processing of the compressed point cloud information generation unit 22. As shown in the figure, the compressed point cloud information generation unit 22 divides the three-dimensional space into an octave tree structure (OCT) as a hierarchical structure composed of multiple nodes, and compresses the point cloud data of the object stored in each node of the octave tree structure.

[0045] In this embodiment, the compressed point cloud information, which includes this compressed point cloud data as part of its data structure, is stored in the compressed point cloud information storage unit 23 shown in Figure 3.

[0046] Figure 5 is a diagram illustrating the schematic structure of the compressed point cloud information stored in the compressed point cloud information storage unit 23. As shown in the figure, the compressed point cloud information consists of a header, metadata, node data, and point cloud data, which are management information.

[0047] Figure 6 is a schematic diagram illustrating the structure of the header, metadata, node data, and point cloud data. As shown in the figure, the header consists of a magic number, version, metadata offset, node data offset, and point cloud data offset.

[0048] In this embodiment, the magic number is a format identifier that identifies the file format of the compressed point cloud information, and the version is data indicating the version of the file format of the compressed point cloud information.

[0049] In this embodiment, the metadata offset is data that stores the address of the starting position of the metadata within the compressed point cloud information, the node data offset is data that stores the address of the starting position of the node data within the compressed point cloud information, and the point cloud data offset is data that stores the address of the starting position of the point cloud data within the compressed point cloud information.

[0050] As shown in Figure 6, in this embodiment, the metadata is data relating to the specifications of the compressed point cloud information, and consists of the number of points, region size, root interval, and accuracy.

[0051] In this embodiment, the score is data that stores the number of points in the point cloud data; the region size is data relating to the region of the point cloud data measured in three-dimensional space (maximum and minimum values ​​in the up, down, left, right, front, and back directions, etc.); the root interval is data relating to the size of the nodes when the three-dimensional space is divided into an octave structure; and the precision is data relating to the precision of the point cloud data.

[0052] As shown in Figure 6, in this embodiment, node data consists of ID, start offset, size, and number of points.

[0053] In this embodiment, ID is data relating to an identification number assigned to each node in the octvine structure, start offset is data storing the address of the starting position of the point cloud data stored in the node, size is data relating to the size of the point cloud data stored in the node, and number of points is data relating to the number of points in the point cloud data stored in the node.

[0054] This node data is constructed by concatenating the same number of data points as the number of nodes in the octvine structure.

[0055] As shown in Figure 6, in this embodiment, the point cloud data consists of data length and binary data. The data length is data relating to the length of the data in the point cloud data, and the binary data is the binary data after the point cloud data received by the input receiving unit 21 has been compressed.

[0056] In this embodiment, the output unit 24 shown in Figure 3 performs the process of outputting the point cloud data of the compressed point cloud information generated by the compressed point cloud information generation unit 22 and stored in the compressed point cloud information storage unit 23 to the user terminal 30 based on a request from the user 2 via the user terminal 30.

[0057] Figure 7 is a block diagram illustrating the general functions of the user terminal 30. As shown in the figure, the user terminal 30 comprises a three-dimensional image processing unit 31, a point cloud data selection unit 32, a point cloud data acquisition unit 33, a point cloud data storage unit 34, a point cloud data adjustment unit 35, and a drawing unit 36.

[0058] The three-dimensional image processing unit 31, point cloud data selection unit 32, point cloud data acquisition unit 33, point cloud data adjustment unit 35, and drawing unit 36 ​​are realized by executing a program stored in memory 102 or software stored in storage 103 on the processor 101, and the point cloud data storage unit 34 is realized by partitioning the storage area of ​​memory 102 or storage 103.

[0059] In this embodiment, the point cloud data selection unit 32, point cloud data acquisition unit 33, point cloud data adjustment unit 35, and drawing unit 36 ​​are configured to function in conjunction with the three-dimensional image processing unit 31.

[0060] In this embodiment, the three-dimensional image processing unit 31 performs various processes related to the movement of three-dimensional images (including both moving and still images) based on the user 2's operations, and functions, for example, as a so-called game engine.

[0061] In this embodiment, the point cloud data selection unit 32 performs the process of selecting point cloud data in three-dimensional space requested by user 2, that is, point cloud data in three-dimensional space within a range that matches user 2's field of view.

[0062] In this embodiment, the point cloud data selection unit 32 functions as a software-based camera that simulates a physical camera, i.e., a "virtual camera." That is, the point cloud data selection unit 32 functions as a virtual camera and performs the process of selecting point cloud data within a range in three-dimensional space that is displayed on the display.

[0063] In this embodiment, the point cloud data acquisition unit 33 performs the process of searching for the point cloud data in the three-dimensional space selected by the point cloud data selection unit 32 from the compressed point cloud information stored in the compressed point cloud information storage unit 23 of the information processing device 20.

[0064] Specifically, the metadata and node data are referenced based on the metadata offset and node data offset of the compressed point cloud information header, and the referenced node data is stored in memory 102.

[0065] Next, the point cloud data acquisition unit 33 searches for point cloud data corresponding to the point cloud data selected by the point cloud data selection unit 32, based on the start offset and size of the node data stored in the memory 102.

[0066] When the point cloud data is searched for, the point cloud data acquisition unit 33 acquires the compressed point cloud data from the information processing device 20 and stores the acquired point cloud data in the point cloud data storage unit 34.

[0067] In this embodiment, the point cloud data adjustment unit 35 performs a process to arbitrarily adjust the dimensions of each point that makes up the acquired point cloud data. In some cases, it automatically adjusts to a predetermined arbitrary dimension, while in other cases, the user 2 manually adjusts to an arbitrary dimension using a coefficient.

[0068] In this embodiment, the drawing unit 36 ​​performs the process of displaying image information based on the point cloud data stored in the point cloud data storage unit 34 on the display of the user terminal 30.

[0069] The drawing process is executed through the processing performed by the point cloud data selection unit 32, point cloud data acquisition unit 33, point cloud data storage unit 34, point cloud data adjustment unit 35, and drawing unit 36.

[0070] Next, an overview of the processing of the information processing system 10 according to this embodiment will be described.

[0071] Figure 8 is a flowchart illustrating the general processing of the information processing system 10. As shown in the figure, in step S1, point cloud data measured in three-dimensional space is uploaded to the information processing device 20. In this embodiment, for example, business operator 1 uploads point cloud data that has been measured and noise removed to the information processing device 20.

[0072] When the point cloud data is uploaded to the information processing device 20 and the information processing device 20 accepts the point cloud data, in step S2, the accepted point cloud data is compressed and compressed point cloud information is generated (compressed point cloud information generation process).

[0073] In this embodiment, the compressed point cloud information generation process divides the three-dimensional space into an octree structure, which is a hierarchical structure composed of multiple nodes, and compresses the point cloud data of the object stored in each node of the octree structure.

[0074] On the other hand, when user 2 selects point cloud data in three-dimensional space via user terminal 30, in step S3, the corresponding point cloud data is searched for in the compressed point cloud information of the information processing device 20.

[0075] When the corresponding point cloud data is searched, the information processing device 20 outputs the point cloud data to the user terminal 30. In step S4, the user terminal 30 acquires the point cloud data and stores the acquired point cloud data in the point cloud data storage unit 34.

[0076] Next, an overview of the drawing process of the information processing system 10 according to this embodiment will be described.

[0077] Figure 9 is a flowchart illustrating the general outline of the drawing process of the information processing system 10. As shown in the figure, first, in step S10, user 2 selects point cloud data in three-dimensional space that user 2 requests via user terminal 30.

[0078] In this embodiment, the system performs a process to select point cloud data for a range in three-dimensional space displayed on the user terminal 30's screen that matches the user 2's field of view. Point cloud data is selected, for example, by pressing an icon displayed on the screen.

[0079] When point cloud data is selected, in step S11, the header of the compressed point cloud information stored in the compressed point cloud information storage unit 23 of the information processing device 20 is read, and in step S12, the metadata and node data are referenced based on the metadata offset and node data offset of the header.

[0080] Next, in step S13, the referenced node data is stored in memory 102, and in step S14, the point cloud data corresponding to the selected point cloud data is searched based on the start offset and size of the node data stored in memory 102.

[0081] When point cloud data is searched, in step S15, compressed point cloud data is obtained from the information processing device 20 and stored in the point cloud data storage unit 34. In step S16, image information based on the point cloud data stored in the point cloud data storage unit 34 is drawn on the display of the user terminal 30.

[0082] In this way, by compressing point cloud data measured in three-dimensional space and generating compressed point cloud information, even large-capacity point cloud data can be easily handled, such as being saved or transferred.

[0083] Furthermore, in this embodiment, by representing the compressed point cloud data as a hierarchical structure, the point cloud data can be clearly distinguished in three-dimensional space, making it easy for user 2 to search for the desired point cloud data.

[0084] Therefore, large amounts of point cloud data can be used efficiently.

[0085] Furthermore, in this embodiment, by selecting point cloud data in a three-dimensional space within a range that matches the user 2's field of view, metadata and node data are referenced based on the header of the compressed point cloud information, and point cloud data is searched based on the referenced node data. As a result, only point cloud data within the range required by user 2 in three-dimensional space can be searched and obtained.

[0086] Therefore, even large-capacity point cloud data can be read quickly and efficiently, significantly reducing the workload for user 2.

[0087] The present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention.

[0088] In the above embodiment, when searching for point cloud data, the metadata and node data are referenced based on the header of the compressed point cloud information, and the point cloud data is searched based on the referenced node data. However, if the compressed point cloud information is stored in the user terminal 30, it can be read using random read.

[0089] In the above embodiment, the case in which the information processing device 20 is implemented on a computer logically realized in a cloud environment was described. However, for example, the information processing device 20 may be implemented on a computer managed by the business operator 1. [Explanation of symbols]

[0090] 1 business operator 2 users 10. Information Processing Systems 20 Information Processing Devices 30. User terminal (terminal)

Claims

1. A compressed point cloud information generation process generates compressed point cloud information represented as a hierarchical structure by encoding the coordinates of the vertices of point cloud data measured in three-dimensional space and irreversibly compressing the point cloud data. A drawing process that searches for the point cloud data in the three-dimensional space requested by the user from the compressed point cloud information generated in the compressed point cloud information generation process, and draws image information based on the retrieved point cloud data on the user's terminal, An information processing system that performs [this action].

2. The aforementioned drawing process is, The image information based on the point cloud data within the range corresponding to the user's field of view in the three-dimensional space is drawn on the user's terminal. The information processing system according to claim 1.

3. The compressed point cloud information is, It comprises management information of the hierarchical structure, metadata, node data relating to the nodes of the hierarchical structure, and compressed point cloud data. The aforementioned drawing process is, Based on the management information, the metadata and node data are referenced, and the point cloud data is searched based on the referenced node data. The information processing system according to claim 1 or 2.

4. The compressed point cloud information is, The node has node data that includes a start offset relating to the starting position of the point cloud data compressed and stored in the node of the hierarchical structure, and the size of the point cloud data stored in the node. The aforementioned drawing process is, The point cloud data is searched based on the start offset and the size. The information processing system according to claim 1 or 2.

5. The compressed point cloud information generation process is as follows: The point cloud data is compressed to generate compressed point cloud information that represents the octvine structure as the hierarchical structure. The information processing system according to claim 1 or 2.

6. The aforementioned drawing process is, The dimensions of the points constituting the point cloud data are arbitrarily adjusted, and the image information based on the point cloud data with adjusted point dimensions is drawn on the user's terminal. The information processing system according to claim 1 or 2.

7. A compressed point cloud information generation process is performed to generate compressed point cloud information, which is represented as a hierarchical structure by encoding the coordinates of the vertices of point cloud data measured in three-dimensional space and irreversibly compressing the point cloud data. The compressed point cloud information generated by the compressed point cloud information generation process is The system comprises management information for the hierarchical structure, metadata, node data relating to the nodes of the hierarchical structure, and compressed point cloud data. Information processing device.

8. The system performs a rendering process that searches for the point cloud data in the three-dimensional space requested by the user from compressed point cloud information, which is obtained by encoding the coordinates of the vertices of point cloud data measured in three-dimensional space, irreversibly compressing the point cloud data, and representing it as a hierarchical structure, and then renders image information based on the retrieved point cloud data. The aforementioned drawing process is, The image information is drawn based on the point cloud data within the range that coincides with the user's field of view in the three-dimensional space. Terminal.

9. Information processing devices implemented by computers, A compressed point cloud information generation process is performed to generate compressed point cloud information, which is represented as a hierarchical structure by encoding the coordinates of the vertices of point cloud data measured in three-dimensional space and irreversibly compressing the point cloud data. A terminal implemented by a computer different from the aforementioned computer, The process involves searching for the point cloud data in the three-dimensional space that the user requests from the compressed point cloud information generated in the compressed point cloud information generation process, and then executing a drawing process to draw image information based on the retrieved point cloud data to the user's terminal. Information processing methods.

Images