Point cloud data display device, point cloud data storage device, point cloud data display method, and point cloud data display program

Abstract

The present invention makes it possible to download and rapidly display large-scale three-dimensional (3D) point group data which is on a point group data distribution server. A point group data display device according to the present invention is provided with a hierarchical storage area, a storage control unit, and a display unit. The hierarchical storage area hierarchizes and stores 3D point group data measured for each among a plurality of continuous unit spaces such that higher levels include 3D point group data included in lower levels. For each of the levels in the hierarchical storage area, the storage control unit acquires, from the storage area one level below, the minimum requisite 3D point group data to be stored in each level of the storage area, and stores the same. The display unit, in accordance with the viewpoint position and line of sight direction of a user, reads the 3D point group data stored in the highest level of the storage area in the hierarchical storage area, and displays said 3D point group data in a 3D virtual reality space.

Description

Technical Field

[0001] One aspect of the present invention relates to a point cloud data display device, a point cloud data storage device, a point cloud data display method, and a point cloud data display program.

Background Art

[0002] In recent years, attempts have been made to view a three-dimensional (3D) virtual reality (VR) space represented in the form of a point cloud through a viewing terminal from a first-person perspective and move within the 3D VR space. As the viewing terminal, it is assumed that a PC (Personal Computer), a smartphone, an HMD (Head Mounted Display), etc. are used. Viewing 3D point cloud data on such a viewing terminal has attracted attention as a new method of spatial expression that provides a different experience from the 3D spaces represented by polygons and the like in the past.

[0003] As a method of having a large number of users view such 3D point cloud data, for example, server-client type data distribution can be considered. That is, 3D point cloud data is arranged on a point cloud data distribution server, and the viewing terminal is used as a client terminal, and the 3D point cloud data is downloaded from the point cloud data distribution server to the viewing terminal via a network such as the Internet.

[0004] When presenting a 3D VR space represented by a point cloud to a user, if the arrangement density in the spatial resolution of the point cloud becomes low, there will be too much space between points, and the reality in the spatial expression will decrease. On the other hand, if an attempt is made to represent a 3D VR space within a certain range with spatially high-resolution and detailed 3D point cloud data, it is not uncommon for the number of point clouds required for spatial expression to reach hundreds of millions to tens of billions of point clouds, and the total data volume will also become an enormous amount of information of several GB to several TB.

[0005] When the number of point clouds increases, that is, when the amount of data becomes large, the following three problems occur in the viewing mode of viewing large-scale 3D point cloud data on the point cloud data distribution server through the viewing terminal.

[0006] The first problem is that when trying to draw a large number of point clouds as a 3DVR space at once on the viewing terminal, it exceeds the graphic performance of the GPU (Graphics Processing Unit) etc. equipped on the viewing terminal, and it is impossible to ensure a drawing frame rate that can withstand viewing. Generally, it is said that the drawing frame rate required to provide a comfortable VR experience is 120fps. For a point cloud data display device with poor graphic performance, it is difficult in terms of the amount of computational processing to continuously draw a large amount of 3D point cloud data at a high frame rate such as 120fps according to the viewing position / angle in the 3DVR space.

[0007] The second problem is that when trying to store a large amount of data in the point cloud data display device at once, it exceeds the data area such as memory and storage that can hold data on the viewing terminal.

[0008] The third problem is that when trying to download the large-scale 3D point cloud data on the point cloud data distribution server to the client terminal via the network from the server, due to the large amount of data, it takes time until the download is completed on the viewing terminal which is the client terminal.

[0009] To solve such problems, various proposals have been made.

[0010] For example, in Non-Patent Document 1, when viewing the 3DVR space with an HMD, a method is proposed to reduce the number of point clouds to be drawn at once by searching for the neighboring points of the 3DVR space in the 360-degree direction from the viewpoint position in the 3DVR space and deleting the points other than the nearest neighbors.

[0011] In Non-Patent Document 2, a method for viewing a large-scale 3D VR space via a web browser is proposed. The method proposed in Non-Patent Document 2 stores the 3D VR space in a state where it can be represented by multiple levels of detail (LoD) of point clouds. For the area that enters the viewing angle of the viewpoint and is in the vicinity of the viewpoint position, a high-density point cloud is referenced, and for the area that enters the viewing angle of the viewpoint and is far away, a low-density point cloud is referenced. This reduces the number of point clouds to be rendered simultaneously at one time, achieving a reduction in the amount of calculation, which is the first problem. Also, in this method, the number of point clouds to be loaded is managed to be below a certain number, and unnecessary point clouds are discarded, thus also solving the second problem.

Prior Art Documents

Non-Patent Documents

[0012]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0013] Although the method proposed in Non-Patent Document 1 can reduce the amount of calculation, which is the first problem, when the viewpoint position is moved, re-search for the nearest point is required each time it is moved, so it is difficult to apply under conditions of frequent movement. Also, this method assumes that all 3D point cloud data is stored in the local storage of the client terminal, and Non-Patent Document 1 does not describe solutions to the second and third problems.

[0014] In the method proposed by Non-Patent Document 2, regarding the third problem, when changing the position or direction of the viewpoint in the 3DVR space, it is necessary to reload the point cloud from the server via the network according to the position or direction of the viewpoint. Therefore, this method has a problem that, until the loading of the 3D point cloud data is completed, although it is a point in the vicinity of the field of view, it is drawn with a low-density point cloud, and there is room for improvement.

[0015] This invention has been made paying attention to the above circumstances, and an object thereof is to provide a point cloud data display device, a point cloud data storage device, a point cloud data display method, and a point cloud data display program that enable large-scale 3D point cloud data on a point cloud data distribution server to be downloaded and displayed at high speed on a viewing terminal that is a client terminal.

Means for Solving the Problem

[0016] In order to solve the above problems, a point cloud data display device according to an aspect of the present invention includes a plurality of hierarchical Second storage areas, a storage control unit, and a display unit. The plurality of hierarchical Second storage areas store, in a hierarchical manner, 3D point cloud data measured for each of a plurality of continuous unit spaces such that the 3D point cloud data included in the lower layer is included in the upper layer. The storage control unit It has a first storage area for storing the second, and controls the 3D point cloud data to be stored in each second storage area according to the viewpoint position and line-of-sight direction of the user, which is obtained from a server capable of communicating via a network. is Control the 3D point cloud data to be stored in each second storage area according to the viewpoint position and line-of-sight direction of the user . The display unit Multiple reads out the 3D point cloud data stored in the storage area of the topmost layer among several hierarchical Second storage areas and displays it in the 3D virtual reality space.

Effects of the Invention

[0017] According to one aspect of the present invention, by adopting a configuration of a multi-level memory area and selecting and loading point cloud information necessary for natural drawing for the user based on the viewpoint position and the line-of-sight direction, it is possible to reduce the constraints on the graphic processing ability of the viewing terminal which is a client terminal, and the influence of the data acquisition time due to the network bandwidth and delay, and enable the large-scale 3D point cloud data on the point cloud data distribution server to be downloaded and displayed at high speed on the viewing terminal, and it is possible to provide a point cloud data display device, a point cloud data storage device, a point cloud data display method, and a point cloud data display program.

Brief Description of Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Mode for Carrying Out the Invention

[0019] [First Embodiment] Hereinafter, with reference to the drawings, the first embodiment according to the present invention will be described.

[0020] (Configuration Example) FIG. 1 is a schematic diagram showing an example of a point cloud data display device according to the first embodiment of the present invention. The point cloud data display device is one in which a large number of viewing terminals 1 and a point cloud data distribution server 2 are connected via a network NW such as the Internet.

[0021] The point cloud data distribution server 2 is a server computer that holds 3D point cloud data to be distributed to a large number of users, and is assumed to have a storage area capable of holding all 3D point cloud data. The 3D point cloud data is obtained by scanning a target facility to be displayed in the 3DVR space with a laser scanner or the like. In the present embodiment, the method for acquiring the 3D point cloud data and the method for transferring the acquired 3D point cloud data to the point cloud data distribution server 2 are not limited in any way.

[0022] Each viewing terminal 1 is a client terminal that downloads and displays 3D point cloud data from this point cloud data distribution server 2.

[0023] FIG. 2 is a block diagram showing an example of the hardware configuration of the viewing terminal 1. The viewing terminal 1 is realized by a computer such as a PC, a smartphone, an HMD, etc. The viewing terminal 1 has, for example, a processor 11A such as a CPU (Central Processing Unit). The processor 11A may be a multi-core / multi-threaded one and can execute a plurality of processes in parallel. And the viewing terminal 1 has a program memory 11B, a data memory 12, a data storage 13, an input / output interface 14, and a communication interface 15 connected to the processor 11A via a bus 16.

[0024] The program memory 11B uses, as a storage medium, a combination of a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written and read at any time and a non-volatile memory such as a ROM (Read Only Memory). The program memory 11B stores programs necessary for the processor 11A to execute various processes. The programs include an OS (Operating System) and various application programs, and as one of the application programs, include the point cloud data display program according to the first embodiment.

[0025] The data memory 12 is a memory area capable of holding general-purpose memory or data for graphic processing. The data memory 12 uses, as a storage medium, a volatile memory such as a RAM (Random Access Memory), for example.

[0026] The data storage 13 is a storage area capable of retaining data. As a storage medium, the data storage 13 uses a non-volatile memory that can be written to and read from at any time, such as an HDD or an SSD. The data storage 13 is used to store the 3D point cloud data downloaded from the point cloud data distribution server 2.

[0027] The input / output interface 14 is an interface between the viewpoint position controller 17 as an input device and the rendering monitor 18 as an output device.

[0028] The viewpoint position controller 17 is a controller for a user who views the 3DVR space represented by the point cloud to control the viewpoint position and the line-of-sight direction within the 3DVR space. Specifically, if the viewing terminal 1 is a PC, the viewpoint position controller 17 is a pointing device such as the arrow keys of a keyboard or a mouse. If the viewing terminal 1 is a smartphone, the viewpoint position controller 17 corresponds to operation buttons arranged on the screen. If the viewing terminal 1 is an HMD, the viewpoint position controller 17 corresponds to the terminal direction obtained by a sensor inside the HMD or a controller for the HMD.

[0029] The rendering monitor 18 is a display device that presents 3DVR space information to the user through the screen.

[0030] The communication interface 15 is a wired or wireless communication unit for connecting to the network NW.

[0031] Although not particularly shown, the point cloud data distribution server 2 can also have a hardware configuration similar to that of the viewing terminal 1 shown in FIG. 2. However, the input devices connected to the input / output interface are a keyboard and a pointing device. Furthermore, the input devices can include a reader for reading files and data to be stored in the data memory from a memory medium such as a USB memory, and a disk device for reading such files and data from a disk medium. The output devices connected to the input / output interface can include a display for displaying output data from the processor, a printer for printing the same, and the like.

[0032] FIG. 3 is a block diagram showing the functional configurations of the viewing terminal 1 and the point cloud data distribution server 2 in association with the hardware configuration shown in FIG. 2. FIGS. 4A and 4B are block diagrams showing the information flow in the functional configurations of the viewing terminal 1 and the point cloud data distribution server 2. In this embodiment, an example of a three-layer storage area of a server, a local storage, and a local memory will be described.

[0033] The processing unit 11 of the viewing terminal 1 is composed of the above-mentioned processor 11A and the above-mentioned program memory 11B. As a software processing functional unit for VR viewing of large-scale 3D space point cloud data on the point cloud data distribution server 2, it includes a viewpoint position acquisition processing unit 111, a point cloud rendering processing unit 112, a local memory position monitoring processing unit 113, a local memory map update processing unit 114, a local storage position monitoring processing unit 115, and a local storage map update processing unit 116. These processing units are all realized by causing the processor 11A to execute the point cloud data display program stored in the program memory 11B. The processing unit 11 may also be realized in the form of other various hardware circuits including integrated circuits such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (field-programmable gate array), and GPU.

[0034] Also, the storage area of the data memory 12 of the viewing terminal 1 includes a spatial position / direction database (hereinafter abbreviated as DB) 121, a local memory point cloud requester 122, a local memory point cloud map 123, a local memory point cloud status 124, a local memory read area map 125, a local storage point cloud requester 126, a local storage point cloud status 127, and a local storage read area map 128.

[0035] Also, the data storage 13 of the viewing terminal 1 includes a local storage point cloud map 131.

[0036] On the other hand, the processing unit 21 of the point cloud data distribution server 2 is also composed of a processor and a program memory, and includes a server point cloud data distribution processing unit 211 as a software-based processing functional unit. This server point cloud data distribution processing unit 211 is realized by causing the processor to execute a point cloud data distribution program stored in the program memory. The processing unit 21 of the point cloud data distribution server 2 may also be realized in various other hardware circuit forms including integrated circuits such as ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (field-programmable gate array), and GPU.

[0037] The storage area of the data storage 23 of the point cloud data distribution server 2 includes a server storage point cloud map 231.

[0038] Before explaining each part of the functional configurations of these viewing terminals 1 and the point cloud data distribution server 2, 3D point cloud data will be explained. 3D point cloud data is point cloud information represented by a data structure capable of expressing a plurality of point information existing in one 3DVR space at a plurality of densities.

[0039] Here, an example of a data structure capable of expressing a plurality of point information existing in this one 3DVR space at a plurality of densities will be explained. FIG. 5 is a schematic diagram for explaining the point cloud density in a cubic space in a three-layer storage area. Further, FIG. 6 is a schematic diagram showing the 3DVR space represented by the cubic space of FIG. 5.

[0040] FIG. 5 shows three cube spaces 31 with different point cloud densities (LoD) when a cube space 31 with a constant side length is assumed as a unit space. Here, the point cloud densities are LoD2 > LoD1 > LoD0. The cube space 31 with the highest point cloud density, LoD2, contains all the point clouds existing within that cube space 31. Next, the cube space 31 with a relatively high point cloud density, LoD1, contains the point cloud information obtained by decimating the point cloud information from the LoD2 point cloud information at a certain ratio. The cube space 31 with the lowest point cloud density, LoD0, contains the point cloud information obtained by decimating the point cloud information from the LoD1 point cloud information at a certain ratio.

[0041] By arranging these cube spaces 31 of LoD2, LoD1, and LoD0 in the X, Y, and Z-axis directions of the three-dimensional box space as shown in FIG. 6, it becomes possible to represent a 3DVR space of any size and the point cloud information included in the 3DVR space with a plurality of point cloud densities (in the case of LoD2, LoD1, and LoD0, three densities). Hereinafter, the region that can be expressed by the three XYZ axes arranging the cube spaces 31 including the point clouds shown in FIG. 6 is referred to as a three-dimensional box space coordinate. The cube space 31 existing in the three-dimensional box space coordinate can be represented by integer values of X, Y, and Z, and can uniquely indicate the position in the three-dimensional box space.

[0042] Note that data structures capable of representing a plurality of point information existing in a 3DVR space with a plurality of densities include, in addition to the data structures shown in FIGS. 5 and 6, the Octree-based MPEG Geometry-based PCC standardized by MPEG (Motion Picture Experts Group), which is a group considering international standardization of videos and the like, the Potree format proposed in Non-Patent Document 2, and the like.

[0043] FIG. 7 is a schematic diagram for explaining the position and direction of a viewpoint in a 3DVR space. In the present embodiment, it is assumed to have a viewpoint position 32 and a viewing direction 33 from a single point in the 3DVR space. The viewpoint position 32 and the viewing direction 33 are, for example, coordinate points x user , y user , z user and Euler angles θx_user , θ y_user , θ z_user It can be defined by. Note that in this case, the x, y, and z coordinate values of the 3DVR space can take real values. Also, for the expression methods of this viewpoint position 32 and line-of-sight direction 33, methods using polar coordinates, methods using quaternions, etc. may be used. Note that in this embodiment, the control of the position and / or direction of the viewpoint in the 3DVR space refers to the user controlling this coordinate and / or line-of-sight direction through a viewpoint position controller 17 such as a PC mouse, a position / direction measuring device of an HMD, a software button of a smartphone, or a built-in sensor.

[0044] As a method of displaying the point cloud of the 3DVR space on the viewing terminal 1, which is a device having a rendering monitor 18 such as a PC, a smartphone, an HMD, etc., a method as shown in FIG. 8 can be adopted. FIG. 8 is a schematic diagram for explaining a method of displaying the point cloud of the 3DVR space on the viewing terminal 1. As shown in FIG. 8, assuming a plane of a rendering screen 34 in the line-of-sight direction on the 3DVR space, the point cloud of the 3DVR space is projected onto the intersection of the line segment connecting the viewpoint position 32 and the point 35 in the 3DVR space and the rendering screen 34. At this time, considering a pair of two points 35 in the 3DVR space that are on the 3DVR space and separated by a distance d in the 3DVR space coordinates, the distance between the two projection points 36 projected onto the rendering screen 34 is relatively closer for the pair of two points farther from the viewpoint position 32 than for the pair of two points closer to the viewpoint position 32. That is, even if there is a point cloud with a certain point cloud density in the 3DVR space, a state occurs where the point cloud appears sparser on the rendering screen 34 in the region closer to the viewpoint position 32, and the point cloud appears denser in the region farther from the viewpoint position 32.

[0045] In this embodiment, a data structure capable of expressing a plurality of point information at a plurality of densities is used. Based on the viewpoint position 32 and the line-of-sight direction 33, the nearby point cloud remains dense, while the distant point cloud is thinned by reducing the number of points. By doing so, without significantly degrading the sense of point cloud density on the rendering screen 34, the number of point clouds to be processed in graphic processing is reduced compared to performing calculations for the entire 3DVR space with a dense point cloud, thus solving the problem of the computational processing amount, which is the first issue. Also, a storage area consisting of three layers, namely a server, a local storage, and a memory for graphic processing, is used to appropriately control the storage range and storage location of the point cloud having a data structure expressed at a plurality of densities based on the viewpoint position 32 and the line-of-sight direction 33 in the 3DVR space. By doing so, the problem of the excess of the data area in the local storage and the memory, which is the second issue, and the problem that the time required to download data from the server becomes large, which is the third issue, are solved.

[0046] FIG. 9 is a schematic diagram showing the relationship between the storage range and storage location of point clouds in a three-layer memory area and the viewpoint position 32 and the line-of-sight direction 33 in the 3DVR space. That is, FIG. 9 shows the data storage 23 of the point cloud data distribution server 2, which is a three-layer memory area, the data storage 13 of the viewing terminal 1, and the data memory 12, the storage range and storage location of the point cloud having a data structure represented by a plurality of densities, and the relationship between the viewpoint position 32 and the line-of-sight direction 33 in the 3DVR space. The data memory 12, which is a memory for graphics, generally has a limited storage capacity and has a smaller storage area than the data storage 23, which is a server storage, and the data storage 13, which is a local storage. Also, if a large amount of point cloud information is read into the data memory 12 for display calculation, the amount of calculation becomes too large. Therefore, it is desirable to read point cloud information in a narrow area and at a low point cloud density in the data memory 12 within a range that does not significantly impair the viewing sensation. Therefore, in the present embodiment, as shown in FIG. 9, the area (storage range and storage location) in the vicinity of the viewpoint position 32 and the line-of-sight direction 33 has a high point cloud density (LoD2), and the areas that are far away or cannot be visually recognized have a low point cloud density (LoD1, LoD0, or no point cloud reading). On the other hand, the data storage 23, which is a server storage, generally easily secures a huge storage capacity. Therefore, the data storage 23 stores data with all point cloud densities (LoD0 to LoD2) in all areas.

[0047] At this time, if an attempt is made to directly read a limited number of point clouds from all the point clouds into the data memory 12, which is the graphics memory of the viewing terminal 1, from the data storage 23 of the point cloud data distribution server 2 within a range that does not impair the viewing sensation, a third problem arises. That is, generally, access between the data storage 23 and the data memory 12 is sandwiched by a network NW such as the Internet and requires a certain amount of time. Therefore, when the viewpoint position 32 and / or the line-of-sight direction 33 moves, the reading cannot keep up, which impairs the viewing sensation of the user.

[0048] Therefore, in this embodiment, a data storage 13, which is a local storage, is sandwiched between the data storage 23 and the data memory 12. Then, also in this data storage 13, in accordance with the viewpoint position 32 and / or the line-of-sight direction 33, by pre-loading point cloud information in a wider range and with higher density than that in the data memory 12, the third problem is solved. Also, generally, although the storage area of the data storage 13 is larger than that of the data memory 12, there are more restrictions than those of the data storage 23 of the point cloud data distribution server 2. Therefore, instead of loading all the point cloud information of all the points into the data storage 13 at once, in accordance with the viewpoint position 32 and / or the line-of-sight direction 33, by loading point cloud information in a wider range than that loaded into the data memory 12 and with lower density than that stored in the data storage 23, the second problem is solved.

[0049] FIG. 10 and FIG. 11 are schematic diagrams for explaining the update of point cloud information in the data memory 12 and the data storage 13 when the viewpoint position 32 and / or the line-of-sight direction 33 in the 3DVR space moves. By loading point cloud information into each of the three-layer storage areas as in this embodiment, even if the viewpoint position and / or the line-of-sight direction in the 3DVR space moves, for example, as shown in FIG. 10, the point cloud on the data memory 12 updates the point cloud information from the data storage 13 faster than that for the point cloud data distribution server 2, and by discarding the point cloud information of the less important part, the storage area of the point cloud information for calculation processing is kept small. Similarly, as shown in FIG. 11, in the data storage 13, the point cloud on the data storage 13 pre-loads, from the point cloud data distribution server 2, the point cloud information in the vicinity of the viewpoint position 32, which is likely to be required by the data memory 12, and in the direction close to the line-of-sight direction 33, up to a point cloud with a higher density as it gets closer to the viewpoint position 32 and the line-of-sight direction 33. By doing so, it becomes possible to pass the point cloud information to the data memory 12 faster than directly accessing the point cloud data distribution server 2 from the data memory 12. Also, in the data storage 13, by performing data reading up to a point cloud with a lower density for the point cloud information far from the viewpoint position 32 and the line-of-sight direction 33, it is realized to keep the storage area of the data storage 13 small.

[0050] Next, each part of the functional configurations of the viewing terminal 1 and the point cloud data distribution server 2 will be described.

[0051] The in-space position and direction DB 121 of the viewing terminal 1 is used to hold the viewpoint position 32 and the line-of-sight direction 33, which are the position of the user's viewpoint and the central direction in the world coordinates of the 3DVR space. FIG. 12 is a schematic diagram for explaining an example of the storage format of the viewpoint position 32 and the line-of-sight direction 33 in the world coordinates in the in-space position and direction DB 121. The user's viewpoint position 32 and line-of-sight direction 33 in the world coordinates can be expressed, for example, as x user , y user , z user , θ x_user , θ y_user , θ z_user in six variables as follows.

[0052] The local memory point cloud requester 122 of the viewing terminal 1 is used to hold the point cloud area (cube space 31) that needs to be read according to the user's viewpoint position and line-of-sight direction. As shown in FIG. 6, when the entire 3DVR space is represented as a space (stereoscopic box space coordinates) divided into three axes of X, Y, and Z by cube spaces 31 of a certain size, the local memory point cloud requester 122 stores an array list indicating which X, Y, and Z values in the stereoscopic box space coordinates the point cloud information in each cube space 31 belongs to and which LoD level it is associated with. FIG. 13 is a schematic diagram for explaining an example of the storage format of the array list in the local memory point cloud requester 122. As shown in FIG. 13, each divided cube space 31 is represented by the X, Y, and Z values and the LoD value of its stereoscopic box space coordinates.

[0053] The local memory point cloud map 123 of the viewing terminal 1 is used to actually store point cloud information. Specifically, for each cube space 31 that divides the entire 3DVR space, which is represented by three-dimensional box space coordinates, the local memory point cloud map 123 stores information on which cube space 31 stores the point cloud at which LoD level, and all the point cloud information contained within that cube space 31. FIG. 14 is a schematic diagram for explaining an example of the storage format of information regarding each cube space 31 in the local memory point cloud map 123. As shown in FIG. 14, the information regarding each cube space 31 consists of the LoD level of the cube space 31, the box numbers of the X, Y, and Z axes of each cube space 31 in the entire 3DVR space, the world coordinates (x point , y point , z point ) and color information (R, G, B) of each point cloud contained in the cube space 31, and finally, the pointer address of the next cube space 31 for forming an array list.

[0054] The local memory point cloud status 124 of the viewing terminal 1 is used to manage the point cloud areas of the point cloud information stored in the local memory point cloud map 123. Specifically, the local memory point cloud status 124 stores the LoD information of the point cloud read into the local memory point cloud map 123. The local memory point cloud status 124 stores the LoD information of the point cloud read into the local memory point cloud map 123 in a data representation similar to the storage format of the array list in the local memory point cloud requester 122 shown in FIG. 13. The difference between the local memory point cloud status 124 and the local memory point cloud requester 122 is that the local memory point cloud status 124 indicates the LoD information of the point cloud actually read into the local memory point cloud map 123.

[0055] The local memory read area map 125 of the viewing terminal 1 is used to store point cloud areas that need to be read according to the user's viewpoint position and line of sight direction. Specifically, the local memory read area map 125 stores information for uniquely linking which cubic space 31 and which LoD are to be acquired among all cubic spaces 31 divided into the entire 3DVR space when the user takes a specific viewpoint position 32 and line of sight direction 33 in the 3DVR space. In this embodiment, as a simple association method, association is performed in the data format shown in FIG. 15.

[0056] FIG. 15 is a schematic diagram for explaining an example of a storage format for associating the viewpoint position 32 and line of sight direction 33 with the acquired cubic space 31 and LoD in the local memory read area map 125. As shown in FIG. 15, the local memory read area map 125 stores them in a format in which a plurality of sets of the range of the rotation angle θz in the z-axis direction of the line of sight direction 33 in the 3DVR space, the 3D box space coordinate values ​​(X', Y', Z'), and the LoD value are arranged. The data shown in FIG. 15 is the 3DVR space coordinates (x user ,y user ,z user ) is the 3D box space coordinate (X n ,Y n ,Z n ), its 3D box space coordinates (X n ,Y n ,Z n ) in the 3D box space coordinates, the cubic space 31 at a relative distance (X', Y', Z') is acquired at the LoD value shown in LoD'. user ,y user ,z user ) and solid box space coordinates (X n ,Y n ,Z n ) is related to the 3DVR space coordinate system, assuming that both coordinates are in the same right-handed system, the origin is the same, and each axis faces the same direction. When one cubic space side of the 3D box space coordinate system is at a distance α in the 3DVR space coordinate system, (x user / α,y user / α,z user / α)=(Xn , Y n , Z n ) holds and can be easily obtained. However, in the left side of this equation, the digits after the decimal point are truncated.

[0057] The local storage point cloud requester 126 of the viewing terminal 1 is used to hold the point cloud area that needs to be read according to the user's viewpoint position and line-of-sight direction. Specifically, the local storage point cloud requester 126 stores the LoD information of the point cloud to be read into the local storage point cloud map 131. The local storage point cloud requester 126 stores the LoD information of the point cloud to be read into the local storage point cloud map 131 in a data representation similar to the storage format of the array list in the local memory point cloud requester 122 shown in FIG. 13.

[0058] The local storage point cloud status 127 of the viewing terminal 1 is used to manage the point cloud area of the point cloud information stored in the local storage point cloud map 131. Specifically, the local storage point cloud status 127 stores the LoD information of the point cloud actually read into the local storage point cloud map 131. The local storage point cloud status 127 also stores the LoD information of the point cloud actually read into the local storage point cloud map 131 in a data representation similar to the storage format of the array list in the local memory point cloud requester 122 shown in FIG. 13.

[0059] The local storage read area map 128 of the viewing terminal 1 is used to store the point cloud area that needs to be read according to the user's viewpoint position and line-of-sight direction. Specifically, the local storage read area map 128, in the same way as the local memory read area map 125 shown in FIG. 15, stores information for uniquely associating which cube space 31 and which LoD in the entire cube space 31 obtained by dividing the entire 3DVR space are to be the acquisition targets when the user takes a specific viewpoint position 32 and line-of-sight direction 33 in the 3DVR space.

[0060] The local storage point cloud map 131 of the viewing terminal 1 is used to actually store the point cloud information. Specifically, the local storage point cloud map 131 is in the same format as the local memory point cloud map 123 shown in FIG. 14. For each cubic space 31 that divides the entire 3DVR space, which is represented by three-dimensional box space coordinates, the local storage point cloud map 131 stores information on which cubic space 31 stores the point cloud at which LoD level, and all the point cloud information contained within that cubic space 31.

[0061] The server storage point cloud map 231 of the point cloud data distribution server 2 is in the same format as the local memory point cloud map 123 of the viewing terminal 1 shown in FIG. 14. For each cubic space 31 that divides the entire 3DVR space, which is represented by three-dimensional box space coordinates, the server storage point cloud map 231 stores information on which cubic space 31 stores the point cloud at which LoD level, and all the point cloud information contained within that cubic space 31. This server storage point cloud map 231 stores the point cloud information of all LoDs of all cubic spaces 31 from the initial state.

[0062] The data amount of the point cloud information read into the local storage point cloud map 131 and the data amount of the point cloud information read into the local memory point cloud map 123 are determined by the read region map data saved in the format shown in FIG. 15. Therefore, it is desirable to set the local storage read region map 128 to be wider and at a higher LoD than the local memory read region map 125. Also, the range and LoD specified by the local storage read region map 128 need to cover the range and LoD data specified by the local memory read region map 125.

[0063] In this way, in the present embodiment, the data read range into the local memory point cloud map 123 and the local storage point cloud map 131 according to the viewpoint position 32 and the line-of-sight direction 33 is set by the setting data on the memory and storage called the read region map data.

[0064] In addition, the viewpoint position acquisition processing unit 111 of the viewing terminal 1 has a function of acquiring the user's viewpoint position and line-of-sight direction from the viewpoint position controller 17. Specifically, the viewpoint position acquisition processing unit 111 of the viewing terminal 1 acquires from the viewpoint position controller 17, via the input / output interface 14, the user's viewpoint position 32(x user , y user , z user , θ x_user ) and the line-of-sight direction 33(θ y_user , θ z_user ) in the world coordinates. Then, the acquired viewpoint position 32 and line-of-sight direction 33 are stored in the in-space position / direction DB 121 as position / direction control information. Note that if the viewpoint position controller 17 always outputs the viewpoint position 32 and the line-of-sight direction 33, the viewpoint position acquisition processing unit 111 acquires the viewpoint position 32 and the line-of-sight direction 33 at regular intervals that are short enough to present the 3DVR space represented by a point cloud to the user following the change of the user's viewpoint position and line-of-sight direction. Also, if the viewpoint position controller 17 outputs the viewpoint position 32 and the line-of-sight direction 33 in response to the change of the user's viewpoint position and line-of-sight direction, the viewpoint position acquisition processing unit 111 acquires them in response to the reception of the output of the viewpoint position controller 17.

[0065] The point cloud rendering processing unit 112 of the viewing terminal 1 is a display control unit that reads out point cloud information from the local memory point cloud map 123 and causes it to be displayed on the rendering monitor 18 via the input / output interface 14.

[0066] The local memory position monitoring processing unit 113 of the viewing terminal 1 monitors the viewpoint position and line-of-sight direction of the user stored in the in-space position and direction DB 121, and when there is a change in at least one of them, determines which point cloud information of the cube space 31 to download and save in the local memory point cloud map 123. Specifically, the local memory position monitoring processing unit 113 acquires the position and direction information in the 3DVR space from the in-space position and direction DB 121, acquires the read area information from the local memory read area map 125, and recalculates the DL request state based on the acquired position and direction information and read area information. Further, the local memory position monitoring processing unit 113 acquires the download (hereinafter abbreviated as DL) request state from the local memory point cloud requester 122, and if the recalculated DL request state is different from this DL request state, updates the local memory point cloud requester 122 according to the recalculated DL request state.

[0067] The local memory map update processing unit 114 of the viewing terminal 1 has a function of acquiring point cloud information from the local storage point cloud map 131 based on the DL request state of the local memory point cloud requester 122 and storing it in the local memory point cloud map 123. Also, during this download, the local memory map update processing unit 114 has a function of preventing unnecessary downloads by checking the local memory point cloud status 124 that stores the download status of the point cloud information in the local memory point cloud map 123. Further, the local memory map update processing unit 114 also has a function of updating the local memory point cloud status 124 when storing the downloaded point cloud information in the local memory point cloud map 123. Specifically, the local memory map update processing unit 114 acquires the DL request state from the local memory point cloud requester 122, acquires the DL state to be checked from the local memory point cloud status 124, and calculates the point cloud area and level information of the difference between these DL request state and DL state. Also, the local memory map update processing unit 114 determines whether there is a difference between the set of elements that exist in the DL request state but not in the DL state and the set of elements that exist in the DL state but not in the DL request state among the calculated difference point cloud area and level information. If there is a difference between these sets of elements, the local memory map update processing unit 114 acquires the DL state from the local storage point cloud status 127, and based on the point cloud area and level information that exists in this DL state among the calculated difference point cloud area and level information, acquires point cloud information from the local storage point cloud map 131. Also, the local memory map update processing unit 114 updates the list of the local memory point cloud map 123 according to the acquired point cloud information. At this time, the local memory map update processing unit 114 may delete the point cloud of the local memory point cloud map 123 based on the calculated difference point cloud area and level information. Further, the local memory map update processing unit 114 updates the DL state of the local memory point cloud status 124 to match the updated point cloud information of the local memory point cloud map 123.

[0068] The local storage position monitoring processing unit 115 of the viewing terminal 1 monitors the viewpoint position and the line-of-sight direction of the user stored in the in-space position and direction DB 121, and when there is a change in at least one of them, determines which point cloud information of the cube space 31 to download and save in the local storage point cloud map 131. Specifically, it acquires the position and direction information in the 3DVR space from the in-space position and direction DB 121, acquires the read area information from the local storage read area map 128, and recalculates the DL request state based on the acquired position and direction information and the read area information. Further, the local storage position monitoring processing unit 115 acquires the DL request state from the local storage point cloud requester 126, and if the recalculated DL request state is different from this DL request state, updates the local storage point cloud requester 126 according to the recalculated DL request state.

[0069] The local storage map update processing unit 116 of the viewing terminal 1 has a function of acquiring point cloud information from the server storage point cloud map 231 of the point cloud data distribution server 2 based on the DL request state of the local storage point cloud requester 126 and storing it in the local storage point cloud map 131. Also, during this download, the local storage map update processing unit 116 has a function of preventing unnecessary downloads by checking the local storage point cloud status 127 that stores the download status of the point cloud information in the local storage point cloud map 131. Further, the local storage map update processing unit 116 also has a function of updating the local storage point cloud status 127 when storing the downloaded point cloud information in the local storage point cloud map 131. Specifically, the local storage map update processing unit 116 acquires the DL request state from the local storage point cloud requester 126, acquires the DL state to be checked from the local storage point cloud status 127, and calculates the point cloud area and level information of the difference between these DL request state and DL state. Also, the local storage map update processing unit 116 determines whether there is a difference between the set of elements that exist in the DL request state but not in the DL state and the set of elements that exist in the DL state but not in the DL request state among the calculated point cloud area and level information of the difference. If there is a difference between these sets of elements, the necessary data is individually requested in units of the cubic space 31 from the server point cloud data distribution processing unit 211 of the processing unit 21 of the point cloud data distribution server 2 via the network NW by the communication interface 15, and the point cloud area and level information and all point cloud information of the cubic space 31 stored in the server storage point cloud map 231 are received. Also, the local storage map update processing unit 116 adds the acquired point cloud area and level information and all point cloud information of one cubic space 31 to the list of the local storage point cloud map 131, and updates the DL state of the local storage point cloud status 127 based on the point cloud area and level information.The local storage map update processing unit 116 repeats the download of point cloud area / level information and all point cloud information from the server storage point cloud map 231 in units of this cubic space 31, the addition of the downloaded data to the local storage point cloud map 131, and the update of the DL state of the local storage point cloud status 127 until all necessary data is downloaded. If all necessary data has been downloaded, the local storage map update processing unit 116 may delete the point cloud of the local storage point cloud map 131 based on the calculated differential point cloud area / level information. In addition, the local storage map update processing unit 116 updates the DL state of the local storage point cloud status 127 so as to match the updated point cloud information of the local storage point cloud map 131.

[0070] Note that each processing unit 111 to 116 of the viewing terminal 1 according to the present embodiment can be realized by a processor 11A and a point cloud data display program stored in advance in a program memory 11B. However, it is also possible to record this point cloud data display program on a non-transitory computer-readable medium or provide it to the viewing terminal 1 through a network. The point cloud data display program thus provided can be stored in the program memory 11B. Further, the provided point cloud data display program can be stored in the data storage 13 which is a storage and executed by the processor 11A as needed, so that the processor 11A can function as the processing units 111 to 116.

[0071] In addition, the server point cloud data distribution processing unit 211 of the point cloud data distribution server 2 receives a necessary data request from the local storage map update processing unit 116 of the viewing terminal 1, and acquires point cloud information based on the necessary data request from the server storage point cloud map 231. Further, the server point cloud data distribution processing unit 211 transmits the acquired point cloud information to the local storage map update processing unit 116.

[0072] This server point cloud data distribution processing unit 211 can also be realized by a processor and a point cloud data distribution program pre-stored in a program memory. However, it is also possible to record this point cloud data distribution program on a non-transitory computer-readable medium or provide it to the point cloud data distribution server 2 through a network. The point cloud data distribution program provided in this way can be stored in the program memory. In addition, the provided point cloud data distribution program can be stored in the data storage 23 which is a storage and executed by the processor as needed, so that the processor can function as the server point cloud data distribution processing unit 211.

[0073] (Operation) Next, the operations of the viewing terminal 1 and the point cloud data distribution server 2 configured as described above will be described.

[0074] FIG. 16 is a flowchart showing an example of the operation processing procedure of the point cloud rendering processing unit 112 of the viewing terminal 1. The processor 11A of the viewing terminal 1 can perform the operation processing shown in this flowchart by executing, for example, a point cloud data display program pre-stored in the program memory 11B. The operation processing shown in this flowchart is started when the point cloud data display program is executed as an independent thread and continuously loop-executed in a form that restarts after the operation processing ends.

[0075] The point cloud rendering processing unit 112 refers to all the point cloud information of all the cubic spaces 31 included in the local memory point cloud map 123 (step S11201). The point cloud information to be referred to is the position coordinates and RGB information in the 3DVR space.

[0076] Subsequently, the point cloud rendering processing unit 112 refers to the position and direction information in the 3DVR space from the in-space position / direction DB 121 (step S11202). The position and direction information to be referred to is the viewpoint position 32 (x user , y user , z user ) and the line-of-sight direction (θx_user ,θ y_user ,θ z_user ).

[0077] Next, the point cloud rendering processor 112 converts the VR space coordinates of all point clouds from the position / direction information and the local memory point cloud map 123 into screen coordinates / camera viewpoint coordinates (step S11203). Specifically, the point cloud rendering processor 112 converts the viewpoint position 32 and line of sight direction 33 (x user ,y user ,z user ,θ x_user ,θ y_user ,θ z_user When a virtual viewpoint camera is placed at the point 35 in each 3DVR space acquired in step S11201, the position information (x point ,y point ,z point ) is projected onto the screen coordinate system (u,v) corresponding to the viewpoint camera coordinate system (x',y',z'), point ,v point In this case, the point cloud rendering processing unit 112 calculates (x point ,y point ,z point ) z' axis value of the viewpoint camera coordinates of point 35 point The relationship in the coordinate transformation of this processing operation is shown in Figure 17.

[0078] FIG. 17 is a schematic diagram for explaining the coordinate conversion of a point arranged in the 3DVR space into the screen coordinate system u, v of the rendering screen 34 corresponding to the viewpoint position 32 and line of sight direction 33 in the 3DVR space. The two axis planes of the screen coordinate system u, v are the same as the plane defined by z'=1 in the viewpoint camera coordinate system x', y', z'. The coordinates (u max ,y max ) is a constant that is uniquely determined by the number of pixels and the viewing angle of the rendering monitor. The origin of the viewpoint camera coordinate system x', y', z' is the viewpoint position (x user ,yuser , z user ) and the line-of-sight direction 33 in the 3DVR space obtained in the above step S11202 coincides with the z'-axis of the viewpoint camera coordinates. The coordinates (u max / 2, y max / 2) on the drawing screen 34 of the z'-axis are the intersection point 37 of the viewpoint camera coordinate system x', y', z' and the screen coordinate system u, v. At this time, the position information (x point , y point , z point ) of each point group obtained in the above step S11201 in the 3DVR space, the projection point 36 (u point , v point ) on the drawing screen 34 and z' point can be calculated by a method that is generally widely known in computer graphics and can be obtained by matrix operations.

[0079] Next, the point cloud rendering processing unit 112 sets the color information (RGB information) of each pixel of the drawing screen based on the calculation results of all the point clouds coordinate-transformed in the above step S11203 for all the pixels of the screen coordinates (step S11204). Specifically, the point cloud rendering processing unit 112 checks for each pixel whether there is a point 35 in the 3DVR space where z' point >0 after being coordinate-transformed to be projected onto that pixel. If there is a corresponding point 35 in the 3DVR space, the point 35 with the smallest z' point is selected, and the RGB information corresponding to that point 35 is used as the color information to be drawn on that pixel. If there is no corresponding point, the RGB information of that pixel is set to (0, 0, 0).

[0080] Then, based on the setting results of the color information (RGB information) for each pixel of the screen coordinates in the above step S11204, the point cloud rendering processing unit 112 sends the drawing information to the drawing monitor 18 (step S11205). After the sending is completed, the point cloud rendering processing unit 112 ends the operation processing procedure shown in this flowchart and starts the operation processing of the above step S11201 again.

[0081] FIG. 18 is a flowchart showing an example of the operation processing procedure of the local memory position monitoring unit 113 of the viewing terminal 1. The processor 11A of the viewing terminal 1 can perform the operation processing shown in this flowchart by executing, for example, a point cloud data display program stored in advance in the program memory 11B. The operation processing shown in this flowchart is started when the point cloud data display program is executed as an independent thread, and is continuously loop-executed in a form that restarts after the operation processing ends.

[0082] The local memory position monitoring unit 113 acquires position and direction information in the 3DVR space from the in-space position and direction DB 121 (step S11301). The position and direction information to be acquired is the viewpoint position 32 (x user , y user , z user ) and the line-of-sight direction (θ x_user , θ y_user , θ z_user ) in the 3DVR space.

[0083] Also, the local memory position monitoring unit 113 acquires the DL request state from the local memory point cloud requester 122 (step S11302). The information to be acquired is the integer values of the X, Y, and Z axes indicating each cubic space 31 of the stereoscopic box space coordinates and a list of the LoD of the cubic space 31. Hereinafter, for the sake of explanation, the number of lists is set to p, and the acquired data is expressed as (X1, Y1, Z1, LoD1),..., (X p , Y p , Z p , LoD p ).

[0084] Furthermore, the local memory position monitoring unit 113 acquires read area information from the local memory read area map 125 (step S11303). Specifically, the local memory position monitoring unit 113 uses the value of θ z_user of the viewing direction acquired in step S11301 to acquire a list of values of X’, Y’, Z’ and LoD’ corresponding to this value of θ z_user .

[0085] Then, the local memory position monitoring processing unit 113 recalculates the DL request state from the position and orientation information in the 3DVR space obtained in the above step S11301 (step S11304). Specifically, the local memory position monitoring processing unit 113 first calculates the three-dimensional box space coordinates (X user / α, y user / α, z user / α) = (X n , Y n , Z n )(where the decimal part is truncated on the left side) based on the position coordinates (x user , y user , z user ) obtained in the above step S11301, to which the position coordinates belong. Next, the local memory position monitoring processing unit 113 calculates (X'1 + X n , Y'1 + Y n , Z'1 + Z n , LoD'1),..., (X' i , Y' i , Z' i , LoD' i ) for the list of i X', Y', Z' and LoD' obtained in the above step S11303, (X'1, Y'1, Z'1, LoD'1),..., (X' n , Y'1 + Y n , Z'1 + Z n , LoD'1),..., (X' i + X n , Y' i + Y n , Z' i + Z n , LoD' i ). At this time, (X'1 + X n , Y'1 + Y n , Z'1 + Z n , LoD'1),..., (X' i + X n , Y' i + Y n , Z' i + Z n , LoD' i ) becomes the recalculated DL request state.

[0086] Here, the local memory position monitoring processing unit 113 uses the DL request status (X’1+X n , Y’1+Y n , Z’1+Z n , LoD’1),…,(X’ i +X n , Y’ i +Y n , Z’ i +Z n , LoD’ i ) recalculated in the above step S11304 and the DL request status (X1, Y1, Z1, LoD1),…,(X p , Y p , Z p , LoD p ) obtained in the above step S11302 to determine whether they match (step S11305). If they match, the local memory position monitoring processing unit 113 ends the operation processing procedure shown in this flowchart and starts the operation processing of the above step S11301 again.

[0087] On the contrary, if it is determined in the above step S11305 that the recalculated DL request status and the obtained DL request status do not match, the local memory position monitoring processing unit 113 updates the local memory point cloud requester 122 with the recalculated DL request status (step S11306). Specifically, the local memory position monitoring processing unit 113 deletes all the values of the local memory point cloud requester 122 and adds the DL request status (X’1+X n , Y’1+Y n , Z’1+Z n , LoD’1),…,(X’ i +X n , Y’ i +Y n , Z’ i +Z n , LoD’ i ) recalculated in the above step S11304 to the local memory point cloud requester 122 to update the local memory point cloud requester 122. After the update is completed, the local memory position monitoring processing unit 113 ends the operation processing procedure shown in this flowchart and starts the operation processing of the above step S11301 again.

[0088] FIG. 19 is a flowchart showing an example of the operation processing procedure of the local memory map update processing unit 114 of the viewing terminal 1. The processor 11A of the viewing terminal 1 can perform the operation processing shown in this flowchart by executing, for example, a point cloud data display program stored in advance in the program memory 11B. The operation processing shown in this flowchart is started when the point cloud data display program is executed as an independent thread, and is continuously loop-executed in a form that restarts after the operation processing ends.

[0089] The local memory map update processing unit 114 acquires the DL request state from the local memory point cloud requester 122 (step S11401). The information to be acquired is a list of integer values of the X, Y, and Z axes indicating each cubic space 31 of the three-dimensional box space coordinates and the LoD of the cubic space 31. Hereinafter, for the sake of explanation, the number of lists is set to p, and the acquired data is expressed as (X mr_1 , Y mr_1 , Z mr_1 , LoD mr_1 ), …, (X mr_p , Y mr_p , Z mr_p , LoD mr_p ).

[0090] Also, the local memory map update processing unit 114 acquires the DL state to be confirmed from the local memory point cloud status 124 (step S11402). The information to be acquired is a list of integer values of the X, Y, and Z axes indicating each cubic space 31 of the three-dimensional box space coordinates and the LoD of the cubic space 31. Hereinafter, for the sake of explanation, the number of lists is set to q, and the acquired data is expressed as (X ms_1 , Y ms_1 , Z ms_1 , LoD ms_1 ), …, (X ms_q , Y ms_q , Z ms_q , LoD ms_q ).

[0091] Then, the local memory map update processing unit 114 calculates the point cloud region and level information of the difference between the DL request state of the acquired local memory point cloud requester 122 and the DL state of the local memory point cloud status 124 (step S11403). Hereinafter, among the calculated difference point cloud region and level information, the DL request state (X mr_1 , Y mr_1 , Z mr_1 , LoD mr_1 ), …, (X mr_p , Y mr_p , Z mr_p , LoD mr_p ) exists, but the DL state (X ms_1 , Y ms_1 , Z ms_1 , LoD ms_1 ), …, (X ms_q , Y ms_q , Z ms_q , LoD ms_q ) does not exist. A set of s elements is represented as (X r_1 , Y r_1 , Z r_1 , LoD r_1 ), …, (X r_s , Y r_s , Z r_s , LoD r_s ). Conversely, a set of u elements that exists in the DL state (X ms_1 , Y ms_1 , Z ms_1 , LoD ms_1 ), …, (X ms_q , Y ms_q , Z ms_q , LoD ms_q ) but does not exist in the DL request state (X mr_1 , Y mr_1 , Z mr_1 , LoD mr_1 ), …, (X mr_p , Y mr_p , Z mr_p , LoD mr_p ) is represented as (X d_1 , Y d_1 , Z d_1 , LoD d_1 ), …, (X d_u , Y d_u , Z d_u , LoD d_u ).

[0092] The local memory map update processing unit 114 determines whether there is a difference in the set of two elements in the differential point cloud region and level information calculated in step S11403, that is, whether s = u = 0 holds (step S11404). If it is determined that there is no difference in the set of two elements, the local memory map update processing unit 114 ends the operation processing procedure shown in this flowchart and starts the operation processing of step S11401 again.

[0093] On the other hand, if it is determined that there is a difference in the set of two elements, the local memory map update processing unit 114 acquires the DL state to be confirmed from the local storage point cloud status 127 (step S11405). The information to be acquired is the integer values of the X, Y, and Z axes indicating each cubic space 31 of the three-dimensional box space coordinates and the list of LoD of the cubic space 31. Hereinafter, for the sake of explanation, the number of lists is set to v, and the acquired data is represented as (X ss_1 , Y ss_1 , Z ss_1 , LoD ss_1 ), …, (X ss_v , Y ss_v , Z ss_v , LoD ss_v ).

[0094] Subsequently, the local memory map update processing unit 114 determines whether the differential point cloud region and level information calculated in step S11403 exists in the DL state of the local storage point cloud status 127 acquired in step S11405 (step S11406). Specifically, the set of elements (X r_1 , Y r_1 , Z r_1 , LoD r_1 ), …, (X r_s , Y r_s , Z r_s , LoD r_s ) in the calculated differential point cloud region and level information and the acquired DL state (X ss_1 , Y ss_1 , Z ss_1 , LoDss_1 ),…,(X ss_v ,Y ss_v ,Z ss_v ,LoD ss_v ) is compared with, and the w elements existing in both are calculated as (X r’_1 ,Y r’_1 ,Z r’_1 ,LoD r’_1 ),…,(X r’_w ,Y r’_w ,Z r’_w ,LoD r’_w ).

[0095] Next, the local memory map update processing unit 114 acquires point cloud information from the local storage point cloud map 131 based on the existing point cloud region and level information determined in step S11406 (step S11407). Specifically, the local memory map update processing unit 114 compares the W elements (X r’_1 ,Y r’_1 ,Z r’_1 ,LoD r’_1 ),…,(X r’_w ,Y r’_w ,Z r’_w ,LoD r’_w ) calculated in step S11406 with the numerically identical data of X, Y, Z, and LoD listed for each cubic space 31 of the local storage point cloud map 131, and acquires the point cloud region and level information and all point cloud information (x point_1 ,y point_1 ,z point_1 ,R1,G1,B1),… of each cubic space 31.

[0096] Also, the local memory map update processing unit 114 adds the point cloud region and level information and all point cloud information (x point_1 ,y point_1 ,z point_1 ,R1,G1,B1),… of each cubic space 31 obtained in step S11407 to the list of the local memory point cloud map 123 (step S11408).

[0097] At this time, the local memory map update processing unit 114 deletes the point cloud of the local memory point cloud map 123 based on the differential point cloud region and level information calculated in step S11403 (step S11409). Specifically, the local memory map update processing unit 114 calculates the set of u elements of the differential point cloud region and level information (X d_1 , Y d_1 , Z d_1 , LoD d_1 ), …, (X d_u , Y d_u , Z d_u , LoD d_u ) and the data in which the numerical values of X, Y, Z, and LoD listed for each cubic space 31 of the local memory point cloud map 123 exactly match, and deletes the point cloud region and level information and all point cloud information (x point_1 , y point_1 , z point_1 , R1, G1, B1), … of each cubic space 31. Note that the operation processing of this step S11409 may not be necessary. That is, the point cloud of the local memory point cloud map 123 may or may not be deleted.

[0098] Then, the local memory map update processing unit 114 updates the DL state of the local memory point cloud status 124 so as to match the updated point cloud information of the local memory point cloud map 123 (step S11410). Specifically, the local memory map update processing unit 114 deletes all elements of the local memory point cloud status 124 once. Thereafter, the local memory map update processing unit 114 lists all data sets of the number b of elements of X, Y, Z, and LoD listed for each cubic space 31 of the local memory point cloud map 123 (X mm_1 , Y mm_1 , Z mm_1 , LoD mm_1 ), …, (X mm_b , Y mm_b , Z mm_b , LoD mm_b) is added to the local memory point cloud status 124. In this way, the local memory map update processing unit 114 updates the DL state of the local memory point cloud status 124. If this update process is completed, the local memory map update processing unit 114 ends the operation processing procedure shown in this flowchart and starts the operation processing of the above step S11401 again.

[0099] FIG. 20 is a flowchart showing an example of the operation processing procedure of the local storage position monitoring processing unit 115 of the viewing terminal 1. The processor 11A of the viewing terminal 1 can perform the operation processing shown in this flowchart by executing, for example, a point cloud data display program stored in advance in the program memory 11B. The operation processing shown in this flowchart is started when the point cloud data display program is executed as an independent thread, and is continuously loop-executed in a form that restarts after the operation processing ends.

[0100] The operation processing procedure of the local storage position monitoring processing unit 115 shown in FIG. 20 is the same as the operation processing procedure of the local memory position monitoring processing unit 113 shown in FIG. 18, except that the acquisition source of the DL request state, the acquisition source of the read area information, and the update destination of the DL request state are the local storage read area map 128 and the local storage point cloud requester 126. Therefore, a detailed description of the operation processing procedure of the local storage position monitoring processing unit 115 with reference to FIG. 20 is omitted.

[0101] FIG. 21 is a flowchart showing an example of the operation processing procedure of the local storage map update processing unit 116 of the viewing terminal 1. The processor 11A of the viewing terminal 1 can perform the operation processing shown in this flowchart by executing, for example, a point cloud data display program stored in advance in the program memory 11B. The operation processing shown in this flowchart is started when the point cloud data display program is executed as an independent thread, and is continuously loop-executed in a form that restarts after the operation processing ends.

[0102] In the operation processing procedures of steps S11601 to S11604 and steps S11610 to S11611 in the operation processing procedure of the local storage map update unit 116 shown in FIG. 21, except for the following points, they are the same as the operation processing procedures of steps S11401 to S11404 and steps S11409 to S11410 in the operation processing procedure of the local memory map update unit 114 shown in FIG. 19. The different points are that the acquisition source of the DL request state is the local storage point cloud requester 126, the point cloud deletion target is the local storage point cloud map 131, and the acquisition source of the DL state and the update destination of the DL request state are the local storage point cloud status 127. Therefore, a detailed description of the operation processing procedures of steps S11601 to S11604 and steps S11610 to S11611 is omitted.

[0103] Hereinafter, only the operation processing procedures of steps S11605 to S11609, which are different from the operation processing procedure of the local memory map update unit 114, will be described. Note that the part different from the operation processing procedure of the local memory map update unit 114 is that the local storage map update unit 116 does not check the cache state of the storage (server) one level above itself. This is because it is assumed that all point clouds are present in the server storage point cloud map 231. That is, in the operation procedure of the local storage map update unit 116, there is no operation processing procedure like checking the local storage point cloud status 127 from the local memory map update unit 114. Also, the local memory map update unit 114 has an operation processing procedure of acquiring all point cloud information in the storage area at once and updating the status, while the local storage map update unit 116 has an operation procedure of acquiring and updating point cloud information one by one for each cubic space 31. This is because it is imagined that a large amount of time is required to acquire data between the point cloud data distribution server 2 and the local storage point cloud map 131.

[0104] Specifically, when the local storage map update processing unit 116 determines that there is a difference in the set of two elements in the differential point cloud area and level information in step S11604, it individually requests the necessary data from the server point cloud data distribution processing unit 211 of the processing unit 21 of the point cloud data distribution server 2 via the network NW by the communication interface 15 (step S11605). Specifically, the local storage map update processing unit 116 sends the information of the element (X r_1 ,Y r_1 ,Z r_1 ,LoD r_1 ),…,(X r_s ,Y r_s ,Z r_s ,LoD r_s ) at the head of the list data as a request to the server point cloud data distribution processing unit 211. Here, (X r_i ,Y r_i ,Z r_i ,LoD r_i ),…,(X r_1 ,Y r_1 ,Z r_1 ,LoD r_1 ),…,(X r_s ,Y r_s ,Z r_s ,LoD r_s ) is a set of elements that exist in the DL request status obtained from the local storage point cloud requester 126 but do not exist in the DL status obtained from the local storage point cloud status 127. At this time, i is a register with an initial value of "1" held inside the local storage map update processing unit 116 or in a partial storage area of the data memory 12, and if a request is sent, the numerical value is incremented by "1" each time. After the increment operation of i is completed, the local storage map update processing unit 116 proceeds to the next step S11606.

[0105] Here, the local storage map update processing unit 116 determines whether it can receive data from the server point cloud data distribution processing unit 211 as a response to the request transmitted in step S11605 (step S11606). The received data is the point cloud region and level information (X r_i , Y r_i , Z r_i , LoD r_i ) of one cubic space 31 of the server storage point cloud map 231 that matches the request in step S11605, and all the point cloud information (x point_1 , y point_1 , z point_1 , R1, G1, B1),... in that cubic space 31. If the reception is not completed, the local storage map update processing unit 116 continues the processing of this step S11606. If the reception processing is completed, the local storage map update processing unit 116 proceeds to the next step S11607.

[0106] The local storage map update processing unit 116 adds the point cloud region and level information (X r_i , Y r_i , Z r_i , LoD r_i ) and all the point cloud information (x point_1 , y point_1 , z point_1 , R1, G1, B1),... of one cubic space 31 obtained in step S11606 to the list of the local storage point cloud map 131 (step S11607).

[0107] Subsequently, the local storage map update processing unit 116 updates the DL state of the local storage point cloud status 127 based on the point cloud region and level information corresponding to the added point cloud information (step S11608). Specifically, the local storage map update processing unit 116 adds the elements of the point cloud region and level information (X r_i , Y r_i , Z r_i , LoD r_i ) added in step S11607 to the local storage point cloud status 127.

[0108] Then, the local storage map update processing unit 116 determines whether or not individual requests for all necessary data have been completed (step S11609). Specifically, the local storage map update processing unit 116 checks the value of i it holds and determines whether i = s. When i ≠ s, the local storage map update processing unit 116 returns the process to step S11605 above and proceeds to receive the next necessary data. When i = s, the local storage map update processing unit 116 determines that the individual request for necessary data has been completed and advances the process to the next step S11610.

[0109] FIG. 22 is a flowchart showing an example of the operation processing procedure of the server point cloud data distribution processing unit 211 of the point cloud data distribution server 2. The processor of the point cloud data distribution server 2 can perform the operation processing shown in this flowchart by executing, for example, a point cloud data distribution program stored in advance in a program memory. The operation processing shown in this flowchart is started when the point cloud data distribution program is executed as an independent thread and is continuously loop-executed in a form that restarts after the operation processing ends.

[0110] The server point cloud data distribution processing unit 211 receives a necessary data request from the local storage map update processing unit 116 of the viewing terminal 1 (step S21101). The necessary data request includes the spatial coordinates of the cubic space 31 and the point cloud area-level information (X, Y, Z, LoD) which is LoD.

[0111] Subsequently, the server point cloud data distribution processing unit 211 acquires point cloud information based on the necessary data request received in the above step S21101 from the server storage point cloud map 231 (step S21102). Specifically, the server point cloud data distribution processing unit 211 acquires information that matches the point cloud area / level information (X, Y, Z, LoD) included in the necessary data request from the server storage point cloud map 231. The information to be acquired is the point cloud area / level information (X, Y, Z, LoD) of one cubic space in the server storage point cloud map 231 and all the point cloud information (x point_1 , y point_1 , z point_1 , R1, G1, B1),... contained in that cubic space.

[0112] Then, the server point cloud data distribution processing unit 211 transmits the point cloud area / level information and all the point cloud information acquired in the above step S21102 to the local storage map update processing unit 116 (step S21103). If the transmission is completed, the server point cloud data distribution processing unit 211 ends the operation processing procedure shown in this flowchart and starts the operation processing of the above step S21101 again.

[0113] As described in detail above, the point cloud data display device according to the first embodiment of the present invention includes a storage area having a plurality of layers, a storage control unit, and a display unit. The storage area having a plurality of layers stores, in a hierarchical manner such that the point cloud information in the lower layer is included in the upper layer, the point cloud information which is 3D point cloud data measured for each of a plurality of continuous unit spaces, i.e., cubic spaces 31, in local memory point cloud map 123, local storage point cloud map 131, and server storage point cloud map 231. The storage control unit includes local memory position monitoring processing unit 113, local memory map update processing unit 114, local storage position monitoring processing unit 115, and local storage map update processing unit 116 which, for each of the storage areas having a plurality of layers, acquire the minimum necessary point cloud information to be stored in the storage area of each layer from the storage area of the layer immediately below and store it. The display unit is point cloud rendering processing unit 112 which reads out the point cloud information stored in the storage area of the topmost layer among the storage areas having a plurality of layers according to the viewpoint position and line-of-sight direction of the user and displays it in the 3DVR space. Therefore, according to the point cloud data display device according to the first embodiment, by dividing the storage area for the point cloud information included in each cubic space 31 into a plurality of layers and hierarchically organizing it, and obtaining the minimum necessary point cloud information in each storage area layer from the viewpoint position and line-of-sight direction of the user and acquiring it from the lower storage area layer, it is possible to reduce the calculation amount of rendering, reduce the storage area required for data retention, and reduce communication delay. That is, the point cloud data display device according to the embodiment has a multi-stage storage area configuration and selects and reads the point cloud information necessary for natural rendering for the user based on the viewpoint position and line-of-sight direction. As a result, it becomes possible to download the large-scale 3D point cloud data on the point cloud data distribution server 2 and display it at high speed on the viewing terminal 1 which is the client terminal. Therefore, when viewing the large-scale, spatially high-resolution 3D point cloud data stored on the point cloud data distribution server 2 via a network NW such as the Internet on the viewing terminal 1, it is possible to prevent the viewing experience of the viewer from being impaired by reducing the influence of the data acquisition time due to the constraints of the graphic processing ability of the viewing terminal 1 and the bandwidth and delay of the network NW.

[0114] In addition, the memory control unit acquires the point cloud information to the memory area using different point cloud densities as data structures with different data amounts of the point cloud information included in each cubic space 31. In this way, by defining the data amount of the point cloud information included in each cubic space 31 at multiple levels and managing, acquiring, and storing the point cloud information for each hierarchical memory area, it is possible to further reduce the calculation amount of drawing, the memory area required for data retention, and the communication delay.

[0115] In addition, the memory control unit includes a viewpoint position acquisition processing unit 111, acquires the user's viewpoint position and line-of-sight direction from the viewpoint position controller 17, and determines the point cloud information to be acquired based on the acquired current viewpoint position and line-of-sight direction. Thereby, for example, the point clouds in the vicinity remain dense from the viewpoint position and line-of-sight direction, and the point clouds far away are thinned by reducing the number of point clouds, so that the number of point clouds to be handled in the graphic processing is less than that when calculating the entire 3DVR space with dense point clouds. As a result, the total calculation processing amount of the entire viewing terminal 1 can be reduced. Also, based on the viewpoint position and line-of-sight direction, it is possible to appropriately control the storage range and storage location of the point cloud having a data structure expressed in multiple densities, and solve the problems of the excess of the data area in the local storage point cloud map 131 and the local memory point cloud map 123, and the problem that the time required to download the point cloud information from the server storage point cloud map 231 becomes large.

[0116] Note that the server storage point cloud map 231, which is the lowest-level memory area among the multiple hierarchical memory areas, includes all the point cloud information for all unit spaces and is arranged on the point cloud data distribution server 2 that can be communicated with by the memory control unit via the network NW. As a result, the viewing terminal 1 does not need to store all the point cloud information, and it is possible to reduce the storage capacity and memory capacity. In addition, the viewing terminal 1 can also reduce the amount of calculation for display by reading the point cloud information in a narrow area and with a low point cloud density within a range that does not significantly impair the viewing experience, and accordingly, the required capabilities of the processor 11A can be suppressed. Therefore, the viewing terminal 1 can be provided at low cost.

[0117] In this case, for the local memory point cloud map 123, which is a storage area other than the storage area of the layer one level higher than the lowest layer among the multiple hierarchical storage areas, the storage control unit acquires all of the necessary minimum amount of point cloud information at once from the local storage point cloud map 131, which is the storage area of the layer one level lower. Also, for the local storage point cloud map 131, which is the storage area of the layer one level higher than the lowest layer, the storage control unit acquires the necessary minimum amount of point cloud information from the server storage point cloud map 231, which is the storage area of the lowest layer arranged on the point cloud data distribution server 2, for each cubic space 31. By doing so, the point cloud information that will be necessary for display on the 3DVR in the immediate vicinity can be read from the local storage point cloud map 131 into the local memory point cloud map 123 at once, enabling it to follow the change in the viewer's viewpoint position and / or line-of-sight direction without delay for display. Also, for the point cloud information that is not immediately necessary but may be necessary for display on the 3DVR, it can be pre-downloaded from the server storage point cloud map 231 into the local memory point cloud map 123 for each cubic space 31.

[0118] [Second Embodiment] In the first embodiment, the viewing terminal 1 is assumed to be realized by a single computer. However, the viewing terminal 1 may be realized by a plurality of computers.

[0119] FIG. 23 is a block diagram showing a configuration example of the viewing terminal 1 in the point cloud data display device according to the second embodiment. As shown in FIG. 23, the viewing terminal 1 can be configured by two computers, a point cloud data storage device 1A and a user interface device 1B. For example, the point cloud data storage device 1A can include a processing unit 1A11 having a local memory position monitoring processing unit 113, a local memory map update processing unit 114, a local storage position monitoring processing unit 115, and a local storage map update processing unit 116, a data memory 12, a data storage 13, and a communication interface 15 similar to those in the first embodiment. Further, the user interface device 1B can include, for example, a processing unit 1B11 having a viewpoint position acquisition processing unit 111 and a point cloud rendering processing unit 112, an input / output interface 14, a viewpoint position controller 17, and a rendering monitor 18 similar to those in the first embodiment. The processing unit 1A11 of the point cloud data storage device 1A and the processing unit 1B11 of the user interface device 1B can communicate information by wire or wirelessly.

[0120] Thus, even if the viewing terminal 1 is configured by a plurality of computers, it can operate in the same manner as in the first embodiment and achieve the same effects.

[0121] [Third Embodiment] In the first embodiment, an example of storing point cloud information in a three-layer storage area of a server, a local storage, and a local memory has been described, but the point cloud information may be stored in a storage area of four or more layers. That is, by arranging any number of layers including two processing units, a position monitoring processing unit that monitors the viewpoint position and line-of-sight direction of the user and a map update processing unit that updates the point cloud information of the point cloud map, and four storage functions, a loading area map that stores which point cloud area (cubic space 31) should be loaded according to the viewpoint position and line-of-sight direction of the user, a point cloud requester that holds the required point cloud area, a point cloud map that actually stores the point cloud information, and a point cloud status that manages the point cloud area of the point cloud information stored in the point cloud map, it is possible to adopt a configuration of four or more layers. These additional layers may be arranged at any of the front stage of the local storage, the middle of the local storage and the local memory, and the rear stage of the local memory.

[0122] FIG. 24 is a block diagram showing a configuration example of the viewing terminal 1 and the point cloud data distribution server 2 in the point cloud data display device according to the third embodiment. FIG. 24 shows an example of a four-layer structure in which a layer of local intermediate memory is provided between the local storage and the local memory. Note that in FIG. 24, for simplicity of the drawing, the configurations of the processing unit 11 and the data memory 12 are partially omitted. In this case, the data memory 12 includes a local intermediate memory read area map 1212 that stores a point cloud area (cubic space 31) that needs to be read according to the viewpoint position and line-of-sight direction of the user, a local intermediate memory point cloud requester 129 that holds the required point cloud area, a local intermediate memory point cloud map 1210 that actually stores the point cloud information, and a local intermediate memory point cloud status 1211 that manages the point cloud area of the point cloud information stored in the local intermediate memory point cloud map 1210. Further, the processing unit 11 includes a local intermediate memory position monitoring processing unit 117 that monitors the viewpoint position and line-of-sight direction of the user stored in the in-space position / direction DB 121, and a local intermediate memory map update processing unit 118 that updates the point cloud information in the local intermediate memory point cloud map 1210. When the local intermediate memory layer is added in this way, the viewing terminal 1 will handle point cloud information at four levels including the point cloud density between LoD1 and LoD0.

[0123] Note that when a layer is added between the local storage and the local memory in this way, the local intermediate memory map update processing unit 118, like the local memory map update processing unit 114, will have an operation processing procedure of acquiring the point cloud information in the storage area at once and updating the status. However, when the data acquisition time takes a long time, the local intermediate memory map update processing unit 118 may also have an operation procedure of acquiring and updating the point cloud information one by one, similar to the local storage map update processing unit 116.

[0124] Also, when additional layers are arranged after the local memory, the operation procedure for updating the map of each additional layer is to obtain and update the point cloud information one by one. For the local storage map update processing unit 116, the operation processing procedure is to obtain the point cloud information in the storage area at once and update the status. That is, only the map update processing unit of the last layer, for which it is guaranteed that the data acquisition source has all the point cloud information, has an operation procedure different from that of the other layers.

[0125] Even if it is extended to four or more layers in this way, the same effects as those of the first embodiment can be achieved.

[0126] Needless to say, the point cloud data storage device 1A of the point cloud data display device according to the second embodiment can also be extended to four or more layers as in this third embodiment.

[0127] [Other Embodiments] Note that the present invention is not limited to the above embodiments.

[0128] For example, in the first to third embodiments, the data reading ranges for the local memory point cloud map 123 and the local storage point cloud map 131 according to the viewpoint position 32 and the line-of-sight direction 33 are set by the memory called read area map data and the setting data on the storage. However, instead of using such memory, storage, and setting data, the range to be read may be geometrically specified. For example, a cubic space 31 enclosed by a sphere with a fixed radius from the viewpoint position 32 is set as a reading target with a specified numerical LoD, and the LoD numerical values of the cubic space 31 to be read are set in multiple levels according to the radius size, and the radius size is set with a wider range and a higher LoD in the local storage point cloud map 131 than in the local memory point cloud map 123, etc. Alternatively, a method may be considered in which a cone is defined with the viewpoint position 32 as the vertex and the center of a circle on the extension line of the line-of-sight direction 33 as the center of the bottom surface, and a unit space enclosed by the cone is set as a reading target with a specified numerical LoD. In the case of a cone, the LoD numerical values of the unit space to be read may be set in multiple levels according to the size of the bottom surface and / or the height of the cylinder, and the set values may be changed between the local storage and the local memory.

[0129] Also, in the first to third embodiments, it is assumed that the point cloud of LoD0 is included in LoD1 and the point cloud of LoD1 is included in LoD2. However, at each LoD, it may be configured not to include overlapping point clouds. In this case, for example, when the local memory point cloud map 123 stores the point cloud information of LoD0 for a certain point cloud area (cubic space 31), when it becomes necessary to store the point cloud information of LoD1 for that point cloud area, the point cloud information of LoD1 is acquired while retaining the point cloud information of LoD0. Thereby, when displaying the point cloud information at the LoD1 level, the point cloud information of both LoD0 and LoD1 can be displayed.

[0130] Also, the flow of each operation process described with reference to the flowchart is not limited to the described procedure, and the order of some steps may be swapped, some steps may be performed simultaneously in parallel, or the processing content of some steps may be modified.

[0131] In addition, the methods described in each embodiment can be stored as a processing program (software means) to be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy (registered trademark) disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can also be transmitted and distributed through a communication medium. Note that the program stored on the medium side includes a setting program for configuring software means (including not only an execution program but also tables and data structures) to be executed by a computer in the computer. The computer that realizes this device reads the program recorded on the recording medium, and in some cases, constructs software means using the setting program, and executes the above-described processing by being controlled by this software means. Note that the recording medium referred to in this specification includes not only a medium for distribution but also storage media such as a magnetic disk and a semiconductor memory provided inside a computer or in a device connected via a network.

[0132] In short, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. Also, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

Description of Reference Numerals

[0133] 1... viewing terminal 1A... point cloud data storage device 1B... user interface device 2... point cloud data distribution server 11A... processor 11B... program memory 11, 1A11, 1B11, 21... processing unit 12... data memory 13, 23... data storage 14…Input / output interface 15…Communication interface 16…Bus 17…Viewpoint position controller 18…Rendering monitor 31…Cube space 32…Viewpoint position 33…Line-of-sight direction 34…Rendering screen 35…Points in 3DVR space 36…Projection point 37…Intersection of viewpoint camera coordinate system and screen coordinate system 111…Viewpoint position acquisition processing unit 112…Point cloud rendering processing unit 113…Local memory position monitoring processing unit 114…Local memory map update processing unit 115…Local storage position monitoring processing unit 116…Local storage map update processing unit 117…Local intermediate memory position monitoring processing unit 118…Local intermediate memory map update processing unit 121…Spatial position and direction DB 122…Local memory point cloud requester 123…Local memory point cloud map 124…Local memory point cloud status 125…Local memory read area map 126…Local storage point cloud requester 127…Local storage point cloud status 128…Local storage read area map 129…Local intermediate memory point cloud requester 1210…Local intermediate memory point cloud map 1211…Local intermediate memory point cloud status 1212…Local intermediate memory read area map 131…Local storage point cloud map 211…Server point cloud data distribution processing unit 231…Server storage point cloud map NW… Network

Claims

1. A first storage area for storing three-dimensional (3D) point cloud data measured for each of a plurality of consecutive unit spaces, and the 3D point cloud data obtained from a server communicable via a network is hierarchically stored such that the 3D point cloud data including lower layers is included in upper layers, a second storage area having a plurality of layers; a storage control unit that controls the 3D point cloud data to be stored in each second storage area according to the viewpoint position and line-of-sight direction of the user; a display unit that reads out the 3D point cloud data stored in the storage area of the topmost layer among the second storage areas having the plurality of layers and displays it in a 3D virtual reality space; A point cloud data display device comprising:

2. The point cloud data display device according to claim 1, wherein the storage control unit acquires the 3D point cloud data into the second storage area using a data structure in which the amounts of data of the 3D point cloud data included in each unit space are different.

3. The point cloud data display device according to claim 1 or 2, wherein the first storage area includes all 3D point cloud data for all unit spaces.

4. The storage control unit For the second storage areas other than the bottommost layer among the second storage areas, all of the 3D point cloud data is acquired at once from the second storage area of the layer one level lower, For the second storage area of the bottommost layer, the 3D point cloud data is acquired for each unit space from the first storage area arranged on the server. The point cloud data display device according to any one of claims 1 to 3.

5. A first storage area for storing three-dimensional (3D) point cloud data measured for each of a plurality of consecutive unit spaces, and the 3D point cloud data obtained from a server communicable via a network is hierarchically stored such that the 3D point cloud data including lower layers is included in upper layers, a second storage area having a plurality of layers; a storage control unit that controls the 3D point cloud data to be stored in each second storage area according to the viewpoint position and line-of-sight direction of the user; A point cloud data storage device comprising:

6. A point cloud data display method in a point cloud data display device that displays 3D point cloud data corresponding to a user's viewpoint position and line-of-sight direction in a three-dimensional virtual reality space, A first storage area for storing three-dimensional (3D) point cloud data measured for each of a plurality of consecutive unit spaces, and storing the 3D point cloud data obtained from a server capable of communicating via a network in a second storage area of a plurality of hierarchies in a hierarchical manner such that the 3D point cloud data including a lower hierarchy is included in a higher hierarchy. Controlling the 3D point cloud data to be stored in each second storage area according to the viewpoint position and line-of-sight direction of the user. Reading out the 3D point cloud data stored in the storage area of the topmost hierarchy among the second storage areas of the plurality of hierarchies and displaying it in a three-dimensional virtual reality space. A point cloud data display method including the above.

7. A point cloud data display program for causing a computer to execute the processing by each part of the point cloud data display device according to any one of Claims 1 to 4.

Images