Information processing device and information processing method

Abstract

An information processing device 1 according to one aspect of the present invention includes: a storage unit 20 that stores 3D point cloud data, which is aggregated data of point data including coordinates and color information within a virtual space; an acquisition unit 11 that acquires parameters including the position of a virtual viewpoint which is the viewpoint of an operator within the virtual space, and a reference display radius of a reference display target point, from among a plurality of display target points that are to be displayed when an image is output, that is a predetermined distance from the virtual viewpoint; a calculation unit 12 that sets a coordinate system for the virtual space and calculates central coordinates and a display radius of each display target point in the virtual space so that the display target point becomes larger as the distance from the virtual viewpoint increases; a generation unit 13 that determines color information for each display target point on the basis of the calculated central coordinates and display radius of each display target point and the 3D point cloud data, and generates display target point data; and a display processing unit 14 that generates an image of the virtual space, as seen from the virtual viewpoint, on the basis of the position of the virtual viewpoint and the display target point data.

Description

Information processing device and information processing method

[0001] One aspect of the present invention relates to an information processing device and an information processing method.

[0002] In recent years, technologies for constructing virtual spaces within computers, such as VR (Virtual Reality), have been developing. Generally, VR spaces are constructed as unrealistic virtual spaces within computer servers using CG (Computer Graphics) technology, etc., which many people can log into. However, in addition to this, there has been a recent trend to create 3D virtual spaces that faithfully reproduce real spaces based on data measured from the real spaces.

[0003] One method for measuring the real world and constructing a virtual space that mimics the real world is to use 3D point cloud data. 3D point cloud data is a collection of point information in a virtual space that has attribute information such as three-dimensional coordinate values ​​(X, Y, Z) and color information (R, G, B), and can be acquired using the LiDAR (Light Detection and Ranging) method.

[0004] "Launch of the 'TENGUN Ogijima Project' to Promote Regional Co-creation", [online], [Retrieved May 23, 2024], Internet,<URL:https: / / group.ntt / jp / newsrelease / 2022 / 11 / 15 / 221115b.html>

[0005] Because 3D point cloud data is three-dimensional information at the time of measurement, it has the advantage of being less expensive than constructing a three-dimensional space from CG or photographs. However, 3D point cloud data often has a huge amount of data. Therefore, unless the specs of the image processing device are high, it takes a very long time to arrange the 3D point cloud in the line of sight from a certain point in three-dimensional space and display each point on the screen. Furthermore, changing the viewpoint requires reloading the 3D point cloud data as needed and performing display calculations, which reduces screen responsiveness and other usability issues. To solve this problem, it is necessary to thin out the information on the points that make up the 3D point cloud data.

[0006] In order to display an image using thinned 3D point cloud data without creating a sense of incongruity with the actual scene, it is necessary to set the display size appropriately. That is, when arranging points in the viewing direction from a certain location in 3D space, the point clouds farther from the viewing position appear denser. Therefore, if all point clouds are set to the same display size, setting the display size to a small value will display distant point clouds clearly, but nearby point clouds will be almost invisible due to gaps. Furthermore, setting the display size to a large value will display nearby point clouds clearly, but distant point clouds will be displayed overlapping each other, resulting in reduced visibility.

[0007] This invention has been made in light of the above circumstances, and aims to provide an information processing device and an information processing method that reduce the amount of data processing required for image processing while maintaining the quality of the displayed image of 3D point cloud data.

[0008] An information processing device according to one aspect of the present invention includes: a memory unit that stores 3D point cloud data, which is a collection of point data including coordinates and color information within a virtual space; an acquisition unit that acquires parameters including the position of a virtual viewpoint, which is the viewpoint of an operator within the virtual space, and a reference display radius of a reference display target point that is a predetermined distance from the virtual viewpoint among a plurality of display target points that are to be displayed when an image is output; a calculation unit that sets a coordinate system for the virtual space and calculates, based on the parameters, the center coordinates and display radius of each display target point in the virtual space for each distance from the virtual viewpoint of the display target point such that the display target point becomes larger as the distance from the virtual viewpoint increases; a generation unit that determines color information for each display target point based on the center coordinates and display radius of each display target point calculated by the calculation unit and the 3D point cloud data, and generates display target point data including the center coordinates, display radius, and color information; and a display processing unit that generates an image of the virtual space as viewed from the virtual viewpoint based on the position of the virtual viewpoint and the display target point data.

[0009] According to one aspect of the present invention, it is possible to provide an information processing device and an information processing method that reduce the amount of data processing in image processing while maintaining the quality of the display image of 3D point cloud data.

[0010] FIG. 1 is a diagram schematically illustrating an example of the configuration of an information processing device according to a first embodiment. FIG. 2 is a flowchart illustrating an example of image generation processing of the information processing device according to the first embodiment. FIG. 3 is a diagram illustrating apparent size according to the first embodiment. FIG. 4 is a diagram illustrating an example of calculation processing according to the first embodiment. FIG. 5 is a diagram illustrating a method for determining the color of a display target point according to the first embodiment. FIG. 6 is a diagram illustrating a 3D point cloud model before processing as viewed from a virtual viewpoint according to the first embodiment. FIG. 7 is a diagram illustrating a 3D point cloud model after processing as viewed from a virtual viewpoint according to the first embodiment. FIG. 8 is a diagram illustrating an example of pre-processing according to a second embodiment. FIG. 9 is a flowchart illustrating an example of pre-processing according to the second embodiment. FIG. 10 is a flowchart illustrating an example of image generation processing according to the second embodiment. FIG. 11 is a diagram illustrating an example of pre-processing according to a third embodiment. FIG. 12 is a flowchart illustrating an example of pre-processing according to the third embodiment. FIG. 13 is a flowchart illustrating an example of image generation processing according to the third embodiment.

[0011] Hereinafter, an embodiment of an information processing apparatus and an information processing method according to the present invention will be described with reference to the drawings. First embodiment

[0012] FIG. 1 is a diagram schematically illustrating an example of the configuration of an information processing apparatus according to the first embodiment.

[0013] The information processing device 1 is a computer capable of outputting a virtual space image that mimics the real world based on acquired 3D point cloud data. The information processing device 1 includes a control unit 10, a storage unit 20, an input unit 30, an output unit 40, and a communication unit 50.

[0014] The input unit 30 acquires various pieces of information supplied from outside the information processing device 1. The input unit 30 is operated by an operator of the information processing device 1 and may include, for example, a user interface such as a mouse or keyboard for inputting various pieces of information (for example, information for changing the numerical values ​​of various settings of the information processing device 1, information indicating the operation of the operator (operation information), etc.), various sensors such as a microphone or a touch panel, etc.

[0015] The output unit 40 outputs, for example, visual or auditory information to be presented to the operator of the information processing device 1. The output unit 40 may include, for example, a monitor or the like that visually supplies information to the operator of the information processing device 1. The output unit 40 may also include an audio output means, such as a speaker, that auditorily supplies information to the operator of the information processing device 1.

[0016] The communication unit 50 can transmit information received from the outside to the components within the information processing device 1, and can also transmit information received from the components within the information processing device 1 to the outside. For example, the communication unit 50 can be connected to a management server, a VR terminal, etc. (not shown) via a network such as the Internet so as to be able to communicate with them.

[0017] The storage unit 20 includes, for example, a main storage unit and an auxiliary storage unit. The main storage unit may include, for example, a read-only memory (ROM) and a random-access memory (RAM). The ROM is a non-volatile memory used exclusively for reading data, and can store data and various setting values ​​used by the control unit 10 when performing various processes. The RAM may be used as a so-called work area for temporarily storing data when the control unit 10 performs various processes. The main storage unit in the embodiment is, for example, a RAM, and is used as a memory.

[0018] The auxiliary storage unit is a non-transitory computer-readable storage medium of a computer centered around the control unit 10. The auxiliary storage unit is, for example, an EEPROM (registered trademark) (electric erasable programmable read-only memory), a hard disk drive (HDD), or a solid state drive (SSD). The auxiliary storage unit can store data and programs used by the control unit 10 when performing various processes, data generated by the processes in the control unit 10, various setting values, and the like.

[0019] The storage unit 20 includes a 3D point cloud data storage area 21 and a display target point cloud buffer 22. The 3D point cloud data storage area 21 stores 3D point cloud data. The 3D point cloud data is used to construct a 3D point cloud model that simulates a real space in a virtual space, and is configured as a collection of multiple point data. The point data that constitutes the 3D point cloud data is configured with information on coordinates (X, Y, Z) and colors (R, G, B) in the virtual space. The 3D point cloud data can be acquired, for example, using LiDAR (Light Detection and Ranging). For example, the 3D point cloud data can be acquired from an external device (such as a management server or LiDAR) connected to the communication unit 50 via a network such as the Internet.

[0020] The display target point cloud buffer 22 is a storage area for temporarily storing the generated display target point data. Here, a display target point is a point (sphere) having a volume in virtual space that is to be displayed when an image in virtual space is output. That is, the display target point data includes the coordinates (center coordinates) of the center point of the display target point in virtual space, color information, and a display radius. The display target point cloud buffer 22 holds, for example, a plurality of display target point data generated by the generation unit 13. The display target point data held in the display target point cloud buffer 22 is read out by the display processing unit 14.

[0021] The control unit 10 is typically a central processing unit (CPU) and / or a graphics processing unit (GPU), but may also include a processor such as a microcomputer, a field programmable gate array (FPGA), or a digital signal processor (DSP). The control unit 10 can implement various functions of the information processing device 1 by executing programs such as system software, application software, or firmware stored in the storage unit 20.

[0022] The control unit 10 includes an acquisition unit 11, a calculation unit 12, a generation unit 13, and a display processing unit 14. The acquisition unit 11 acquires various parameters used when generating an image. For example, the acquisition unit 11 accepts parameter input by an operator via the input unit 30. The various parameters include initial parameters, which will be described later. The acquisition unit 11 stores the acquired various parameters in the storage unit 20.

[0023] The calculation unit 12 sets a coordinate system for the virtual space and calculates the position of the display target point from the virtual viewpoint and the display radius of the display target point. The virtual viewpoint indicates the viewpoint of the operator in the virtual space. The calculation unit 12 calculates the center coordinates of the display target point in the virtual space and the display radius of the display target point based on various parameters set in advance by the operator or input via the input unit 30, for example.

[0024] The generation unit 13 generates display target point data. The generation unit 13 determines the color of the display target point based on the 3D point cloud data acquired from the 3D point cloud data storage area 21 and the center coordinates and display radius of the display target point acquired from the calculation unit 12, and generates the display target point data.

[0025] The display processing unit 14 generates an image (still image or video) in the virtual space. The display processing unit 14 generates an image in the virtual space as seen from the virtual viewpoint based on a coordinate system in the virtual space, a virtual viewpoint, and display target point data. The display processing unit 14 outputs the generated image to the operator via the output unit 40 or stores it in the storage unit 20, for example.

[0026] Next, an example of the operation of the information processing device 1 in the first embodiment will be described.

[0027] 2 is a flowchart showing an example of image generation processing of the information processing device in the first embodiment. The control unit 10 of the information processing device 1 executes the following operations in response to a start instruction from the operator.

[0028] The acquisition unit 11 acquires initial parameters (step S1). The calculation unit 12 receives, for example, input of the initial parameters by an operator. Alternatively, the initial parameters may be set in advance by the operator and stored in the storage unit 20, or may be acquired from the outside via the communication unit 50.

[0029] The initial parameters include information about the virtual viewpoint, a rendering limit distance from the virtual viewpoint, a reference display radius, an overlap rate, and a visual diameter value. The virtual viewpoint information includes information about the coordinates of the virtual viewpoint and the viewing direction. The rendering limit distance indicates the limit of the distance that can be rendered from the virtual viewpoint. In the embodiment, the reference display radius indicates the display radius of the display target point closest to the virtual viewpoint. The overlap rate indicates the degree of overlap between a certain display target point and a display target point adjacent to that display target point in the depth direction. In other words, the overlap rate indicates the degree of overlap between two display target points adjacent to that display target point in the depth direction when viewed from the virtual viewpoint.

[0030] The apparent diameter value included in the initial parameters is a parameter that indicates the apparent size of a point when viewed from a virtual viewpoint. For example, when a point of a predetermined size (display radius) is viewed from a virtual viewpoint in a virtual space, even if the point has the same size, the apparent size of the point will be smaller the farther the point is from the virtual viewpoint, and larger the closer the point is from the virtual viewpoint.

[0031] 3 is a diagram illustrating apparent size in the first embodiment. In the embodiment, the value of the apparent diameter of a display target point in a virtual space indicates the relationship between the distance from the virtual viewpoint to the center point of the display target point and the display radius of the point. Specifically, if the value of the distance from the virtual viewpoint to the display target point is d and the value of the display radius of the point is r, the value of the apparent diameter of the point is 2r / d.

[0032] Here, if the value of the visual diameter is constant, the apparent size of the display target point from the virtual viewpoint will also be constant. For example, in Figure 3, the value of the visual diameter of the display target point 1011 with a display radius r1 located at a distance d1 from the virtual viewpoint 100 is 2r1 / d1. Also, the value of the visual diameter of the display target point 1012 with a display radius r2 located at a distance d2 from the virtual viewpoint 100 is 2r2 / d2. Here, if 2r1 / d1 = 2r2 / d2, the apparent sizes of the points 1011 and 1012 will not change.

[0033] The calculation unit 12 sets a coordinate system for the virtual space, and calculates the center coordinates and display radius of each display target point based on the acquired initial parameters (step S2).

[0034] 4 is a diagram illustrating an example of calculation processing in the first embodiment. First, the calculation unit 12 calculates the distance ds of the reference display target point 101s from the virtual viewpoint 100 based on the reference display radius rs and the visual diameter 2r / d included in the initial parameters. The reference display target point is a display target point that serves as a reference when generating display target points. In the embodiment, the reference display target point is the display target point that is closest to the virtual viewpoint.

[0035] The calculation unit 12 calculates the distance ds of the reference display target point 101s from the virtual viewpoint and then determines the position of the reference display target point 101s. For example, the calculation unit 12 arranges multiple reference display target points 101s so that the center coordinates are arranged so that the distance from the virtual viewpoint is ds and adjacent points are in contact with each other, and calculates the center coordinates. Alternatively, the multiple reference display target points may be displayed so that adjacent points partially overlap each other.

[0036] Next, the calculation unit 12 calculates the distance d from the virtual viewpoint 100 and the display radius r of the display target point 101 of the next layer based on the distance ds and the display radius rs of the reference display target point 101s from the virtual viewpoint and the visual diameter 2r / d and overlap rate o included in the initial parameters, and determines the central coordinates of the display target point 103. Here, a layer refers to a point cloud of multiple display target points that are the same distance from the virtual viewpoint to their central coordinates and have the same display radius value. Also, the next layer refers to a layer that is adjacent to a given layer in the depth direction when viewed from the virtual viewpoint.

[0037] The calculation unit 12 arranges the display target point 101 of the next layer so as to overlap the center point (coordinate point) of the reference display target point 101s in the depth direction by a distance obtained by multiplying the display radius rs by the overlap rate o, as viewed from the virtual viewpoint 100. In this embodiment, if the value of the overlap rate o is 100% (1.0), the reference display target point 101s and the display target point 101 of the next layer are arranged so as to overlap by the distance rs.

[0038] The calculation unit 12 calculates the distance from the virtual viewpoint and the display radius of the display target point of the next layer based on the distance from the virtual viewpoint and the display radius value of the display target point of the previous layer, the apparent diameter value, and the overlap rate, and repeatedly performs the process of determining the center coordinates. The calculation unit 12 repeats this process until the distance value of the display target point from the virtual viewpoint reaches a maximum value that does not exceed, for example, the rendering limit distance value. The calculation unit 12 inputs the calculated center coordinates and display radius of each display target point to the generation unit 13.

[0039] The generation unit 13 determines color information for each display target point based on the center coordinates and display radius of each display target point calculated by the calculation unit 12 and the 3D point cloud data stored in the 3D point cloud data storage area 21, and generates display target point data (step S3).

[0040] FIG. 5 is a diagram illustrating a method for determining the color of a display target point in the first embodiment. In a virtual space, a display target point 101 is displayed as a sphere with a display radius r and a coordinate point as its center. The generation unit 13 determines a point 102 based on the 3D point cloud data that exists within the display target point 101 based on the coordinates of each point data included in the 3D point cloud data, and extracts the point data. The generation unit 13 separates the color information included in the extracted multiple point data into R, G, and B, adds them, and divides the average value by the number of extracted point data to determine the color information of the display target point. If no point data exists within the display target point, or if the number of point data within the display target point is below a predetermined threshold, the generation unit 13, for example, deletes the data of the center coordinates and display radius of the display target point, thereby preventing the display target point from being displayed. For example, the predetermined threshold is determined based on the value of the display radius of the display target point.

[0041] When the generation unit 13 determines color information for each display target point whose center coordinates and display radius have been calculated by the calculation unit 12, it generates display target point data that associates the center coordinates, color information, and display radius for each display target point and inputs the data to the display target point group buffer 22.

[0042] The display processing unit 14 outputs an image of the virtual space as seen from the virtual viewpoint based on the coordinates of the virtual viewpoint acquired by the calculation unit 12 and the display target point data (center coordinates, color information, and display radius) stored in the display target point cloud buffer 22 (step S4). For example, the display processing unit 14 receives, via the input unit 30, a designation of the field of view direction from the virtual viewpoint in the virtual space and a change in the field of view direction, and performs image processing such as affine transformation and z-buffering, thereby outputting an image of the virtual space as seen from the virtual viewpoint to the operator via the output unit 40. When the control unit 10 of the information processing device 1 receives an end instruction from the operator, it ends the above image generation process.

[0043] Next, the effects achieved by the information processing device in the first embodiment will be described.

[0044] 6 is a diagram showing a 3D point cloud model before processing as seen from a virtual viewpoint in the first embodiment. The upper part of FIG. 6 shows an example of a 3D point cloud model 110 constructed in a virtual space based on 3D point cloud data stored in the 3D point cloud data storage area 21, and the virtual viewpoint 100. For the sake of explanation, the outline of the 3D point cloud model 110 is shown with a solid line, but in reality, it is drawn as a collection of countless points arranged in the virtual space. Here, it is assumed that each of the points constructing the 3D point cloud model 110 has a constant display radius and arrangement density (number per unit volume).

[0045] The bottom panel of Figure 6 shows the field of view, as seen from the virtual viewpoint 100, of multiple points on a plane represented by a point pattern in the 3D point cloud model 110 in the top panel. Looking at the bottom panel of Figure 6, it can be seen that points closer to the virtual viewpoint appear larger in apparent size than points further back. It can also be seen that points closer to the virtual viewpoint appear with a lower apparent density than points further back. Therefore, if the display size of all point clouds is set to the same value, setting the display size to a small value will display distant point clouds clearly, but nearby point clouds will be barely visible due to gaps. Setting the display size to a large value will display nearby point clouds clearly, but distant point clouds will be displayed overlapping, reducing visibility.

[0046] Fig. 7 is a diagram showing a 3D point cloud model after processing from a virtual viewpoint in the first embodiment. The upper part of Fig. 7 shows an example of a 3D point cloud model 111 constructed in a virtual space based on the display target point data generated by the calculation unit 12 and the generation unit 13, and a virtual viewpoint 100. As in Fig. 6, the outline of the 3D point cloud model 111 is shown by a solid line, but in reality it is drawn as a collection of countless points arranged in the virtual space.

[0047] Here, each of the points (display target points) that make up the 3D point cloud model 111 is displayed with a different display radius depending on the distance from the virtual viewpoint 100, and the density of the point cloud also varies depending on the distance from the virtual viewpoint. That is, points farther from the virtual viewpoint have a larger display radius, and points closer to the virtual viewpoint have a smaller display radius. Furthermore, point clouds farther from the virtual viewpoint are arranged with a lower density, and point clouds closer to the virtual viewpoint are arranged with a higher density.

[0048] The bottom part of Fig. 7 shows the field of view of multiple points on a plane, represented by a dot pattern, of the 3D point cloud model 111 in the top part, as seen from the virtual viewpoint 100. Looking at the bottom part of Fig. 7, the apparent size of the points as seen from the virtual viewpoint is constant regardless of the distance from the virtual viewpoint. Furthermore, the apparent density of the point cloud as seen from the virtual viewpoint does not change significantly depending on the distance from the virtual viewpoint.

[0049] As a result, the image in the virtual space generated by the information processing device in the first embodiment can maintain the visibility of the point cloud regardless of the distance from the viewpoint in the virtual space. Furthermore, in the first embodiment, the density of the point cloud at a position far from the virtual viewpoint is low (the number of points is small), so the amount of calculation can be reduced compared to displaying an image based on the original 3D point cloud data.

[0050] As described above, the information processing device according to the first embodiment calculates the center coordinates and display radius of each display target point in the virtual space for each distance from the virtual viewpoint of the display target point based on parameters including the position of the virtual viewpoint, which is the viewpoint of the operator in the virtual space, and the reference display radius of a reference display target point that is a predetermined distance from the virtual viewpoint among multiple display target points that are to be displayed when an image is output, so that the display target point becomes larger as the distance from the virtual viewpoint increases, determines color information for each display target point based on the calculated center coordinates and display radius of each display target point and the stored 3D point cloud data, generates display target point data including the center coordinates, display radius, and color information, and generates an image of the virtual space. Therefore, in the image generated by the information processing device 1 according to the first embodiment, point clouds that are farther away from the viewpoint in the virtual space have a large display size and a low density, and point clouds that are closer to the viewpoint in the virtual space have a small display size and a high density.

[0051] As a result, the information processing device 1 according to the first embodiment can reduce the amount of data processing in image processing while maintaining the quality of the display image of the 3D point cloud data. Second embodiment

[0052] Next, an information processing apparatus according to a second embodiment will be described. The information processing apparatus according to the second embodiment may have the same configuration as the information processing apparatus according to the first embodiment, and therefore a description thereof will be omitted.

[0053] 8 is a diagram illustrating an example of pre-processing in the second embodiment. Unlike the first embodiment, the second embodiment generates an image in a virtual space when a virtual viewpoint in the virtual space moves along a certain path.

[0054] In the second embodiment, the information processing device 1 performs pre-processing to create display target points in advance for areas far from the virtual viewpoint. Here, as an example, a case will be described in which the virtual space is divided into a short distance area and a medium-to-long distance area according to the distance from the virtual viewpoint for each virtual viewpoint position on the path along which the virtual viewpoint passes, but the method of dividing the virtual space is not limited to this. For example, the information processing device 1 according to the second embodiment may divide the virtual space into a short distance area and a medium-to-long distance area according to the distance from the path along which the virtual viewpoint passes.

[0055] 9 is a flowchart showing an example of pre-processing in the second embodiment. First, the acquisition unit 11 of the information processing device acquires initial parameters (step S101). In the pre-processing, the initial parameters include, for example, a value of the rendering limit distance, a value of the visual diameter, a value of the reference display radius, and an overlap rate. In this example, the reference display radius is set to the value of the display radius of the reference display target point, which is the display target point closest to the position of the virtual viewpoint on the path along which the virtual viewpoint passes, as in the first embodiment.

[0056] The calculation unit 12 sets a coordinate system for the virtual space and a plurality of positions of the virtual viewpoint on the path along which the virtual viewpoint passes, and calculates, for each of the set virtual viewpoint positions, a display radius and center coordinates of a display target point according to the distance from the virtual viewpoint (step S102). The detailed method of calculating the coordinates and display radius of the display target point may be the same as in the first embodiment, and therefore a detailed description thereof will be omitted. For example, the calculation unit 12 sets positions at any unit distance from the starting point of the path along which the virtual viewpoint passes as positions of the virtual viewpoint on the path along which the virtual viewpoint passes.

[0057] The generation unit 13 determines color information for each display target point based on the center coordinates and display radius of each display target point calculated for each of the multiple virtual viewpoint positions by the calculation unit 12 and the 3D point cloud data stored in the 3D point cloud data storage area 21 (step S103). The process by which the display target point generation unit determines color information for the display target points may be the same as in the first embodiment, so a detailed description will be omitted. After determining the color information for each display target point, the generation unit 13 generates display target point data that associates the center coordinates, color information, and display radius of each display target point.

[0058] In the second embodiment, the generation unit 13 associates the generated display target point data with the original 3D point cloud data and the set position of the virtual viewpoint, and stores the data in the storage unit 20 (step S104). After executing the process of step S4, the control unit 10 of the information processing device 1 ends the pre-processing.

[0059] Next, the image generation process executed by the information processing device in the second embodiment will be described.

[0060] 10 is a flowchart showing an example of image generation processing in the second embodiment. First, the acquisition unit 11 acquires parameters such as the rendering limit distance, the visual diameter, the reference display radius, and the overlap rate, as well as the value of the judgment distance (step S105), and then acquires information on the virtual viewpoint, which is the viewpoint of the operator in the virtual space (step S106). The parameters acquired by the acquisition unit 11 may be the same as the parameters acquired during pre-processing.

[0061] Here, the determination distance is a threshold used to divide an area into a close distance area and a medium-to-long distance area. In the embodiment, if the value of the determination distance is D, the calculation unit 12 determines that an area in the virtual space whose distance from the path taken by the virtual viewpoint is less than D is a close distance area. The calculation unit 12 determines that an area in the virtual space whose distance from the path taken by the virtual viewpoint is D or more (less than the rendering limit distance) is a medium-to-long distance area.

[0062] The calculation unit 12 calculates the center coordinates and display radius of the display target points in the short distance area based on the acquired parameters (step S107). For example, the calculation unit 12 calculates the center coordinates and display radius of the display target points whose distance values ​​from the virtual viewpoint are less than the acquired determination distance D. The detailed calculation method of the coordinates and display radius of the display target points may be the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0063] The generation unit 13 determines color information for each of the display target points in the close distance area (step S108) based on the center coordinates and display radius of each of the display target points calculated by the calculation unit 12 and the 3D point cloud data stored in the 3D point cloud data storage area 21. The process by which the generation unit 13 determines color information for the display target points may be the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0064] When the generation unit 13 determines color information for each display target point within the close-range area whose coordinates and display radius have been calculated by the calculation unit 12, it generates display target point data that associates the center coordinates, color information, and display radius, and inputs this data to the display target point cloud buffer 22.

[0065] The display processing unit 14 outputs an image of the virtual space as seen from the virtual viewpoint, based on the virtual viewpoint information acquired by the calculation unit 12, the display target point data in the close distance area held in the display target point cloud buffer 22, and the display target point data in the medium to long distance area generated in pre-processing and stored in the storage unit 20 (step S109). That is, the display processing unit 14 uses the display target point data generated by the calculation unit 12 and the generation unit 13 for the close distance area less than the determination distance from the position of the virtual viewpoint, and generates an image of the virtual space using the display target point data created in advance in pre-processing and stored in the storage unit 20 for the medium to long distance area equal to or greater than the determination distance from the position of the virtual viewpoint.

[0066] The control unit 10 of the information processing device 1 in the second embodiment repeats the processes of steps S106 to S109 until an end instruction is received from the operator (step S110). For example, when the position of the virtual viewpoint is changed based on an input from the operator, the control unit 10 generates display target point data for display target points in the close distance area, and generates and outputs an image of the virtual space seen from the virtual viewpoint after the position change based on the generated display target point data and display target point data that has been generated and stored in advance and that corresponds to the position of the virtual viewpoint after the change.

[0067] As described above, the information processing device according to the second embodiment generates display target point data in advance for areas that are a predetermined distance or more from the virtual viewpoint as a pre-processing step. As a result, even when the virtual viewpoint in the virtual space moves, the information processing device according to the second embodiment only needs to perform calculations for display target points in the close-distance area, eliminating the need to perform calculations for all display target points in the virtual space. This allows the information processing device according to the second embodiment to maintain the visibility of the point cloud regardless of the distance from the viewpoint in the virtual space, and also reduces the amount of data processing required even when the viewpoint position moves. Third embodiment

[0068] Next, an information processing device according to a third embodiment will be described. The information processing device according to the third embodiment may have the same configuration as the information processing devices according to the first and second embodiments, and therefore description thereof will be omitted.

[0069] 11 is a diagram illustrating an example of pre-processing in the third embodiment. In the third embodiment, similarly to the second embodiment, an image in a virtual space is generated when a virtual viewpoint in the virtual space moves along a certain path.

[0070] In the third embodiment, the point clouds are grouped into predetermined areas, and the path of the virtual viewpoint is determined in advance. The information processing device 1 generates display target points for each area in advance as a pre-processing, and uses the previously generated display target points for areas that are a predetermined distance or more from the virtual viewpoint during the image generation process.

[0071] 12 is a flowchart showing an example of pre-processing in the third embodiment. First, the acquisition unit 11 of the information processing device acquires initial parameters (step S201). In the pre-processing, the initial parameters include, for example, a value of the rendering limit distance and a value of the visual diameter.

[0072] The calculation unit 12 sets a coordinate system for the virtual space, calculates the display radius of the display target points for each area previously set in the virtual space, and determines coordinates (step S202). For example, the calculation unit 12 calculates the display radius of the display target points in each area based on the distance from the path along which the virtual viewpoint passes to the center of the area and the apparent diameter. The calculation unit 12 determines the center coordinates of the display target points based on the calculated display radius values. For example, the calculation unit 12 determines the center coordinates of each display target point by arranging the display target points so that they are in contact with each other.

[0073] The generation unit 13 determines color information for each display target point based on the center coordinates and display radius of each display target point calculated by the calculation unit 12 and the 3D point cloud data stored in the 3D point cloud data storage area 21 (step S203). The process by which the display target point generation unit determines color information for the display target points may be the same as in the first embodiment, so a detailed description will be omitted. After determining the color information for each display target point, the generation unit 13 generates display target point data in which the center coordinates, color information, and display radius of each display target point are associated with each other.

[0074] In the third embodiment, the generation unit 13 associates the generated display target point data with the original 3D point cloud data and stores it in the storage unit 20 (step S204). For example, the generation unit 13 stores the generated display target point data by area. After executing the process of step S204, the control unit 10 of the information processing device 1 ends the pre-processing.

[0075] Next, the image generation process executed by the information processing device in the third embodiment will be described.

[0076] 13 is a flowchart showing an example of image generation processing in the third embodiment. First, the acquisition unit 11 acquires an area determination threshold, an initial display radius value, and an overlap rate (step S205), and acquires information on a virtual viewpoint, which is the viewpoint of the operator in the virtual space (step S206).

[0077] The calculation unit 12 determines whether each area in the virtual space is within a predetermined distance from the virtual viewpoint (step S207). The calculation unit 12 calculates the distance from the virtual viewpoint for each area in the virtual space. If the value of the distance from the virtual viewpoint of an area is equal to or less than the area determination threshold acquired in step S205, the calculation unit 12 determines that the area is within the predetermined distance from the virtual viewpoint. If the value of the distance from the virtual viewpoint of an area is greater than the area determination threshold acquired in step S205, the calculation unit 12 determines that the area is not within the predetermined distance from the virtual viewpoint.

[0078] The calculation unit 12 calculates the center coordinates and display radius of the display target point in the area within a predetermined distance from the virtual viewpoint based on the acquired parameters and the initial parameters acquired during pre-processing (step S208). The detailed calculation method of the center coordinates and display radius of the display target point may be the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0079] The generation unit 13 determines color information for each of the display target points in the area within a predetermined distance from the virtual viewpoint (step S209), based on the center coordinates and display radius of each of the display target points calculated by the calculation unit 12 and the 3D point cloud data stored in the 3D point cloud data storage area 21. The process by which the generation unit 13 determines color information for the display target points may be the same as in the first embodiment, and therefore a detailed description thereof will be omitted.

[0080] When the generation unit 13 determines color information for each display target point in an area within a predetermined distance from the virtual viewpoint whose center coordinates and display radius have been calculated by the calculation unit 12, the generation unit 13 generates display target point data that associates the center coordinates, color information, and display radius, and inputs the data to the display target point cloud buffer 22.

[0081] The display processing unit 14 outputs an image of the virtual space as seen from the virtual viewpoint based on the information of the virtual viewpoint acquired by the calculation unit 12, the display target point data of the area within a predetermined distance held in the display target point cloud buffer 22, and the display target point data of the area determined not to be within a predetermined distance from the virtual viewpoint generated in pre-processing and stored in the memory unit 20 (step S210).

[0082] The control unit 10 of the information processing device 1 in the second embodiment repeats the processes of steps S206 to S210 until an end instruction is received from the operator (step S211). For example, when the position of the virtual viewpoint is changed based on an input from the operator, the control unit 10 generates and outputs an image of the virtual space as seen from the virtual viewpoint after the change in position.

[0083] As described above, the information processing device according to the third embodiment generates display target point data in advance for each predetermined area in a virtual space as pre-processing. Then, during image generation processing, the information processing device performs image processing using the pre-generated display target point data for areas that are not within a predetermined distance from the position of the virtual viewpoint. As a result, according to the information processing device according to the third embodiment, even when the virtual viewpoint in the virtual space moves, it is sufficient to perform calculation processing for display target points in areas within the predetermined distance, and there is no need to perform calculations for all display target points in the virtual space. This allows the information processing device according to the third embodiment to maintain the visibility of the point cloud regardless of the distance from the viewpoint in the virtual space, and also reduces the amount of data processing required even when the viewpoint position moves.

[0084] The information processing device 1 described above may be configured as a single server device, or may be configured as multiple server devices connected via a network. For example, the information processing device 1 may be a virtual information processing device (cloud server) configured on a network. Even in this case, the same effects as those of the above-described embodiment can be obtained.

[0085] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0086] REFERENCE SIGNS LIST 1 information processing device 10 control unit 11 acquisition unit 12 calculation unit 13 generation unit 14 display processing unit 20 storage unit 21 3D point cloud data storage area 22 display target point cloud buffer 30 input unit 40 output unit 50 communication unit 100 virtual viewpoint 101, 1011, 1012 display target point 101s reference display target point 102 point 110, 111 3D point cloud model

Claims

1. An information processing device comprising: a memory unit that stores 3D point cloud data, which is a collection of point data including coordinates and color information within a virtual space; an acquisition unit that acquires parameters including the position of a virtual viewpoint, which is the viewpoint of an operator within the virtual space, and a reference display radius of a reference display target point that is a predetermined distance from the virtual viewpoint among a plurality of display target points that are to be displayed when an image is output; a calculation unit that sets a coordinate system for the virtual space and calculates, based on the parameters, the center coordinates and display radius of each of the display target points in the virtual space for each distance from the virtual viewpoint of the display target point such that the display target point becomes larger the farther the distance from the virtual viewpoint; a generation unit that determines color information for each of the display target points based on the center coordinates and display radius of each of the display target points calculated by the calculation unit and the 3D point cloud data, and generates display target point data including the center coordinates, the display radius, and the color information; and a display processing unit that generates an image of the virtual space as seen from the virtual viewpoint based on the position of the virtual viewpoint and the display target point data.

2. The information processing device of claim 1, wherein the parameters further include a visual diameter indicating the relationship between the distance from the virtual viewpoint to the display target point and the display radius of the display target point, and the calculation unit calculates the center coordinates and the display radius of each of the display target points in the virtual space based on the position of the virtual viewpoint, the reference display radius, and the visual diameter.

3. The information processing device of claim 2, wherein the calculation unit calculates the distance from the virtual viewpoint of the reference display target point and the center coordinates based on the reference display radius and the visual diameter, and calculates the distance from the virtual viewpoint of the display target point adjacent to the reference display target point and the display radius in the depth direction from the virtual viewpoint to the center point of the reference display target point so as to satisfy the relationship indicated by the visual diameter.

4. The information processing device of claim 3, wherein the parameters further include an overlap rate indicating the degree of overlap between two adjacent display points in the depth direction when viewed from the virtual viewpoint, and the calculation unit calculates the center coordinates and the display radius of the display point adjacent to any of the display target points so that the relationship indicated by the visual diameter is satisfied in the depth direction from the virtual viewpoint to the center point of any of the display target points, and so that the display radius of the arbitrary display target point overlaps with the arbitrary display target point by a distance equal to the product of the overlap rate and the display radius of the arbitrary display target point.

5. The information processing device according to claim 1, wherein the generation unit extracts point data of a plurality of points existing within each of the display target points based on coordinate information of each of the point data contained in the 3D point cloud data, and determines the average value of the color information contained in the extracted point data as the color information of the display target point.

6. The information processing device of claim 1, wherein the memory unit further stores the display target point data created in advance for areas whose distance from the virtual viewpoint is equal to or greater than a predetermined judgment distance, the calculation unit calculates the center coordinates and the display radius of the display target point whose distance from the virtual viewpoint is less than the judgment distance, and the display processing unit generates the image based on the display target point data stored in the memory unit for areas whose distance from the virtual viewpoint is equal to or greater than the judgment distance.

7. The information processing device of claim 1, wherein the memory unit further stores the display target point data created in advance for each area set in advance in the virtual space, the calculation unit determines for each of the areas whether it is within a predetermined distance from the virtual viewpoint, and calculates the center coordinates and the display radius of the display target point for the areas within the predetermined distance, and the display processing unit generates the image based on the display target point data stored in the memory unit for the areas determined not to be within the predetermined distance from the virtual viewpoint.

8. An information processing method which acquires parameters including the position of a virtual viewpoint, which is the viewpoint of an operator in a virtual space, and a reference display radius of a reference display target point that is a predetermined distance from the virtual viewpoint among a plurality of display target points to be displayed when an image is output; sets a coordinate system for the virtual space, and calculates, based on the parameters, the center coordinates and display radius of each of the display target points in the virtual space for each distance from the virtual viewpoint of the display target point, such that the display target point becomes larger as the distance from the virtual viewpoint increases; determines color information for each of the display target points based on the calculated center coordinates and display radius of each of the display target points and 3D point cloud data, which is a collection of point data including coordinate and color information in the virtual space; generates display target point data including the center coordinates, the display radius, and the color information; and generates an image of the virtual space as seen from the virtual viewpoint based on the position of the virtual viewpoint and the display target point data.

Images