Information processing device and information processing method

Abstract

An information processing device 1 according to one embodiment of the present invention includes: a storage unit 20 that stores 3D point group data, namely, set data of point data including coordinate information and color information in a virtual space; a calculation unit 121 that sets a coordinate system for the virtual space and sets a virtual viewpoint including information regarding the position and direction of the viewpoint of an operator in the virtual space; a distance calculation unit 122 that calculates, for individual pieces of point data included in the 3D point group data, a relative distance between a virtual viewpoint and a point in the virtual space based on the point data in the direction of the virtual viewpoint, and determines whether point data is present in an interpolation target area on the basis of the value of the relative distance and a first threshold value; an interpolation data calculation unit 123 that generates interpolation point data on the basis of the point data of two adjacent points in the virtual space for point data determined to be present in the interpolation target area by the distance calculation unit 122; and a rendering unit 124 that generates an image of the virtual space as viewed from the virtual viewpoint using the 3D point group data, the virtual viewpoint, and the interpolation point data.

Description

Information processing device and information processing method

[0001] One aspect of the present invention relates to an information processing device and an information processing method.

[0002] In recent years, technologies for constructing virtual spaces within computers, such as VR (Virtual Reality), have been developing. Generally, VR spaces are constructed as unrealistic virtual spaces within computer servers using CG (Computer Graphics) technology, etc., which many people can log into. However, in addition to this, there has been a recent trend to create 3D virtual spaces that faithfully reproduce real spaces based on data measured from the real spaces.

[0003] One method for measuring the real world and constructing a virtual space that mimics the real world is to use 3D point cloud data, which is a collection of point information in a virtual space that has attribute information such as three-dimensional coordinate values ​​(X, Y, Z) and color information (R, G, B).

[0004] One method for acquiring 3D point cloud data is the LiDAR (Light Detection and Ranging) method, which measures the distance to an object, direction, and shape of the object by irradiating the object with laser light and measuring the time it takes for the light to bounce back after hitting the object. 3D point cloud data can be acquired by collecting points that represent the results of the laser light scan.

[0005] "The TENGUN Ogijima Project" launched to promote regional co-creation", [online], [Retrieved May 23, 2024], Internet,<URL:https: / / group.ntt / jp / newsrelease / 2022 / 11 / 15 / 221115b.html> Matsumoto Takahiro, Komazaki Kei, Maki Yuichi, Chiaki Hiroshi, Mochizuki Takayoshi, "TENGUN Ogijima: Research on highly immersive VR through visual, auditory, and tactile sensations based on measurement data", [online], March 1, 2023, Information Processing Society of Japan Interaction 2023, [Retrieved May 23, 2024], Internet,<URL:https: / / www.interaction-ipsj.org / proceedings / 2023 / data / pdf / 2B-17.pdf> "What is LiDAR? Explaining the principles and uses of light sensors that are attracting attention in autonomous driving", [online], [Retrieved May 23, 2024], Internet,<URL:https: / / engineer.fabcross.jp / archeive / 211109_lidar.html> "What is point cloud data? | Acquisition methods and usage examples", [online], [Retrieved May 23, 2024], Internet,<URL:https: / / club.informatix.co.jp / ?p=1125>

[0006] 3D point cloud data has the advantage of being three-dimensional at the time of measurement, making it less expensive than constructing a three-dimensional space from CG or photographs. However, the volume of 3D point cloud data can be enormous, and arranging the 3D point cloud in the line of sight from a certain point in three-dimensional space and displaying each point on a screen requires extremely high specs for the image processing device. Furthermore, changing the line of sight while displaying the data requires the 3D point cloud data to be reloaded and displayed again, which means that devices with low specs cannot keep up and the display cannot keep up, reducing usability.

[0007] One way to solve the above problem is to thin out the information about the points that make up the 3D point cloud data, thereby reducing the amount of data and making it possible to process it even on devices with low specifications. However, thinning out the 3D point cloud data causes gaps between points to become noticeable when displayed on a screen, which causes a problem of reduced image quality.

[0008] One way to address the degradation in image quality caused by thinning out 3D point cloud data is to interpolate between gapped points when displaying the data. However, data interpolation requires a certain amount of CPU power, so thinning out the data to enable processing on devices with low specifications contradicts the reduction in data volume.

[0009] This invention has been made in light of the above circumstances, and aims to provide an information processing device and an information processing method that suppress deterioration in apparent image quality when outputting images and display 3D point cloud data with a small amount of data processing.

[0010] An information processing device according to one aspect of the present invention includes: a memory unit that stores 3D point cloud data, which is a collection of point data including coordinate information and color information in a virtual space; a calculation unit that sets a coordinate system for the virtual space and sets a virtual viewpoint including information about the position and direction of an operator's viewpoint in the virtual space; a distance calculation unit that calculates, for each piece of point data included in the 3D point cloud data, a relative distance between the virtual viewpoint and a point in the virtual space based on the point data in the direction of the virtual viewpoint, and determines whether the point data exists within an interpolation target area based on the value of the relative distance and a predetermined first threshold value; an interpolation data calculation unit that generates interpolated point data for point data determined by the distance calculation unit to exist within the interpolation target area based on point data of two adjacent points in the virtual space; and a rendering unit that generates an image of the virtual space as seen from the virtual viewpoint using the 3D point cloud data, the virtual viewpoint, and the interpolated point data.

[0011] According to one aspect of the present invention, it is possible to provide an information processing device and an information processing method that suppress deterioration in the apparent image quality when outputting an image and display 3D point cloud data with a small amount of data processing.

[0012] FIG. 1 is a diagram schematically showing an example of the configuration of an information processing device according to an embodiment. FIG. 2 is a flowchart showing an example of interpolation processing performed by the information processing device according to the embodiment. FIG. 3 is a diagram showing a 3D point cloud model constructed in a virtual space in the example. FIG. 4 is a diagram for explaining the interpolation processing performed by the information processing device according to the example. FIG. 5 is a diagram showing a portion of a 3D point cloud model before interpolation as seen from a virtual viewpoint in the example. FIG. 6 is a diagram showing a portion of a 3D point cloud model after interpolation as seen from a virtual viewpoint in the example. FIG. 7 is a diagram schematically showing an example of the configuration of an information processing device according to a second embodiment. FIG. 8 is a diagram for explaining an example of point cloud recognition processing performed by the information processing device according to the second embodiment.

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 1 is a diagram illustrating an example of the configuration of an information processing apparatus according to an embodiment of the present invention.

[0014] The information processing device 1 is a computer capable of outputting a virtual space image that mimics the real world based on acquired 3D point cloud data. The information processing device 1 includes a control unit 10, a storage unit 20, an input unit 30, an output unit 40, and a communication unit 50.

[0015] The input unit 30 acquires various pieces of information supplied from outside the information processing device 1. The input unit 30 is operated by an operator of the information processing device 1 and may include, for example, a user interface such as a mouse or keyboard for inputting various pieces of information (for example, information for changing the numerical values ​​of various settings of the information processing device 1, information indicating the operation of the operator (operation information), etc.), various sensors such as a microphone or a touch panel, etc.

[0016] The output unit 40 outputs, for example, visual or auditory information to be presented to the operator of the information processing device 1. The output unit 40 may include, for example, a monitor or the like that visually supplies information to the operator of the information processing device 1. The output unit 40 may also include an audio output means, such as a speaker, that auditorily supplies information to the operator of the information processing device 1.

[0017] The communication unit 50 can transmit information received from the outside to the components within the information processing device 1, and can also transmit information received from the components within the information processing device 1 to the outside. For example, the communication unit 50 can be connected to a management server, a VR terminal, etc. (not shown) via a network such as the Internet so as to be able to communicate with them.

[0018] The storage unit 20 includes, for example, a main storage unit and an auxiliary storage unit. The main storage unit may include, for example, a ROM (read-only memory) and a RAM (random-access memory). The ROM is a non-volatile memory used exclusively for reading data, and can store data and various setting values ​​used by the control unit 10 when performing various processes. The RAM may be used as a so-called work area for temporarily storing data when the control unit 10 performs various processes. The main storage unit in the embodiment is, for example, a RAM, and is used as a memory.

[0019] The auxiliary storage unit is a non-transitory computer-readable storage medium of a computer centered around the control unit 10. The auxiliary storage unit is, for example, an EEPROM (registered trademark) (electrically erasable programmable read-only memory), a hard disk drive (HDD), or a solid state drive (SSD). The auxiliary storage unit can store data and programs used by the control unit 10 when performing various processes, data generated by the processes in the control unit 10, various setting values, and the like.

[0020] In the embodiment, the storage unit 20 stores 3D point cloud data 21 and thinned 3D point cloud data 22. The 3D point cloud data 21 is data used to construct a 3D point cloud model that simulates a real space in a virtual space, and is configured as a collection of multiple point data. The point data that constitutes the 3D point cloud data 21 is configured with coordinate information (X, Y, Z) in the virtual space and color information (R, G, B). The 3D point cloud data 21 can be acquired, for example, using LiDAR (Light Detection and Ranging). LiDAR is a device that irradiates a target with laser light and measures the round-trip time of the laser light to measure the distance, direction, position, shape, etc. of an object. For example, the 3D point cloud data 21 is acquired from an external device (such as a management server or LiDAR) connected via a network such as the Internet by the communication unit 50, and stored in the storage unit 20.

[0021] The post-thinning 3D point cloud data 22 is data obtained after a thinning process, which will be described later, is performed on the 3D point cloud data 21. The number of point data constituting the post-thinning 3D point cloud data 22 is smaller than the number of point data constituting the 3D point cloud data 21, and therefore the post-thinning 3D point cloud data 22 has a smaller data volume than the 3D point cloud data 21.

[0022] The control unit 10 is typically a central processing unit (CPU) and / or a graphics processing unit (GPU), but may also include a processor such as a microcomputer, a field programmable gate array (FPGA), or a digital signal processor (DSP). The control unit 10 can implement various functions of the information processing device 1 by executing programs such as system software, application software, or firmware stored in the storage unit 20.

[0023] The control unit 10 includes a point cloud data reduction unit 11 and a thinned point cloud display function unit 12. The point cloud data reduction unit 11 performs a thinning process on the 3D point cloud data 21 to generate thinned 3D point cloud data 22. The point cloud data reduction unit 11 thins out point data included in the 3D point cloud data 21 based on a thinning number setting value stored in the storage unit 20. The thinning number setting value is set in advance, for example, by an operator and stored in the storage unit 20. For example, when the set thinning number setting value is N, the point cloud data reduction unit 11 reduces the number of point data included in the 3D point cloud data 21 by thinning out (N-1) pieces of point data at intervals of N along the coordinate axis (thinning out (N-1) pieces of point data between the kth point and the (N+k)th point). The point cloud data reduction unit 11 stores the thinned 3D point cloud data 22 generated by thinning the 3D point cloud data based on the thinning number setting value in the storage unit.

[0024] The thinned point cloud display function unit 12 performs interpolation processing on the thinned 3D point cloud data 22 to generate an image with reduced degradation in apparent image quality. Detailed processing (interpolation processing) by the thinned point cloud display function unit 12 will be described later. The thinned point cloud display function unit 12 is composed of a calculation unit 121, a distance calculation unit 122, an interpolation data calculation unit 123, and a rendering unit 124.

[0025] The calculation unit 121 sets a virtual space (for example, a coordinate system that serves as a reference for the space). For example, the calculation unit 121 constructs a 3D point cloud model in the virtual space based on the acquired 3D point cloud data. In the embodiment, when the calculation unit 121 acquires the thinned 3D point cloud data 22, the calculation unit 121 constructs a 3D point cloud model in the virtual space by arranging points in the virtual space based on the point data included in the thinned 3D point cloud data 22.

[0026] The calculation unit 121 also controls a virtual viewpoint, which is the viewpoint of the operator in the virtual space. The virtual viewpoint is composed of two pieces of information: a virtual viewpoint position and a virtual viewpoint direction. The virtual viewpoint position is coordinate information of the virtual viewpoint in the virtual space. The virtual viewpoint direction is information indicating the direction in which the virtual viewpoint is facing (the direction of the operator's line of sight). For example, the virtual viewpoint direction includes information on an azimuth angle indicating the angle on the horizontal plane (a plane perpendicular to the direction of gravity) of the virtual space, and an elevation / depression angle indicating the angle on a vertical plane perpendicular to the horizontal plane. Alternatively, the virtual viewpoint direction may be represented by a vector.

[0027] The distance calculation unit 122 calculates the distance between each point constituting the 3D point cloud model constructed in the virtual space and the virtual viewpoint grasped by the calculation unit 121. For each piece of point data constituting the thinned 3D point cloud data 22, the distance calculation unit 122 calculates the relative distance between the two in the virtual space (between the coordinates of the virtual viewpoint and the coordinates of the point data) based on coordinate information indicated by the virtual viewpoint position and coordinate information held by the point data. For example, the distance calculation unit 122 calculates the Euclidean distance between the virtual viewpoint position and the point data.

[0028] Furthermore, the distance calculation unit 122 determines whether or not the point data exists within the interpolation target area based on the calculated value of the relative distance from the point data to the virtual viewpoint and an interpolation area threshold (first threshold). The interpolation area threshold is set in advance, for example, by an operator and stored in the storage unit 20. The distance calculation unit 122 determines that point data whose calculated value of the relative distance from the point data to the virtual viewpoint is equal to or less than the interpolation area threshold is within the interpolation target area.

[0029] Here, multiple interpolation area thresholds may be set by the operator. For example, the distance calculation unit 122 may use M-1 interpolation area thresholds to divide the virtual space into M areas. For example, when the virtual space is divided into M areas, the distance calculation unit 122 can classify each divided area as a first area, a second area, ..., an Mth area in order of proximity to the virtual viewpoint. The distance calculation unit 122 determines, for each piece of point data, whether or not it exists in any of the areas. In this case, the areas from the first area to the M-1th area are the interpolation target areas.

[0030] Alternatively, the distance calculation unit 122 may set the interpolation target area in advance. For example, the distance calculation unit 122 may set the interpolation target area in the virtual space in advance based on the virtual viewpoint position and an interpolation area threshold. The distance calculation unit 122 can determine, for each piece of point data included in the acquired thinned-out 3D point cloud data 22, whether the point data exists within the interpolation target area based on coordinate information held by the point data.

[0031] The interpolation data calculation unit 123 generates a number of interpolation point data based on the thinning number setting value, based on point data of two adjacent points that exist within the interpolation target area. The interpolation data calculation unit 123 determines the number of interpolation point data to generate based on the thinning number setting value stored in the storage unit 20. For example, if the stored thinning number setting value is N, the interpolation data calculation unit 123 generates N-1 pieces of interpolation point data. Furthermore, if multiple interpolation area thresholds are set, that is, if there are multiple interpolation target areas, the interpolation data calculation unit 123 sets the number of interpolation point data to generate for each area.

[0032] The interpolation point data generated by the interpolation data calculation unit 123 is composed of coordinate information and color information, similar to the point data included in the 3D point cloud data 21 and the thinned 3D point cloud data 22. Based on the coordinate information and color information of two point data determined to be adjacent, the interpolation data calculation unit 123 calculates the coordinate information and color information of interpolation point data that interpolates the space between the two adjacent points. For example, based on the coordinate information of the two adjacent point data, the interpolation data calculation unit 123 calculates the coordinate information of the interpolation point data so that the interpolation point data is evenly distributed on the line connecting the two adjacent points. Furthermore, for example, the interpolation data calculation unit 123 sets the average value of each color included in the color information of the two adjacent point data as the color information of the interpolation point data.

[0033] Here, two adjacent points are, for example, a point and a point whose relative distance from the point is equal to or less than the interpolation point distance threshold. The interpolation point distance threshold is, for example, preset by the operator and stored in the storage unit 20. Alternatively, two adjacent points are, for example, a point and a point whose relative distance from the point is equal to or less than the interpolation point distance threshold and that are closest to the point. Alternatively, two adjacent points are, for example, a point and a point that are on a plane that is perpendicular to the virtual viewpoint direction and includes the point (or on a spherical surface that is centered on the virtual viewpoint and tangent to the plane perpendicular to the virtual viewpoint direction and includes the point), and that are adjacent to the point in any direction, or that have the smallest relative distance from the point. Note that the interpolation point distance threshold may be determined according to a thinning number setting value.

[0034] The rendering unit 124 generates an image (still image or video) of the virtual space as seen from the virtual viewpoint. The rendering unit 124 generates an image of the virtual space as seen from the virtual viewpoint based on the virtual space set by the calculation unit 121 and the virtual viewpoint position and virtual viewpoint direction. The rendering unit 124 transmits the generated image to the output unit 40. Alternatively, the rendering unit 124 may store the generated image in the storage unit 20 or transmit it to the outside via the communication unit 50.

[0035] Next, an example of the interpolation process executed by the information processing device 1 will be described. FIG. 2 is a flowchart showing an example of the interpolation process executed by the information processing device in the embodiment. First, as preprocessing, the point cloud data reduction unit 11 executes a thinning process on the 3D point cloud data 21 stored in the storage unit 20. In this example, the thinning number setting value N used by the point cloud data reduction unit 11 in the thinning process is set to 3. That is, in this example, the point cloud data reduction unit 11 thins out two points at intervals of three points. By executing the thinning process on the point cloud data 21, the point cloud data reduction unit 11 generates thinned 3D point cloud data 22 and stores it in the storage unit 20.

[0036] The thinned point cloud display function unit 12 of the information processing device 1 according to the embodiment starts the following interpolation process, for example, based on an operation by an operator. The calculation unit 121 sets a coordinate system and a virtual viewpoint in a virtual space (step S1). The calculation unit 121 acquires the thinned 3D point cloud data 22 stored in the storage unit 20 and constructs a 3D point cloud model in the virtual space. The calculation unit 121 also sets a virtual viewpoint, which is the viewpoint of the operator in the virtual space, based on information input by the operator via the input unit 30, for example.

[0037] 3 is a diagram showing a 3D point cloud model constructed in a virtual space in the embodiment. In the embodiment, the calculation unit 121 sets a coordinate system in the virtual space, with the horizontal direction as the x-axis, the vertical direction (height) as the y-axis, and the depth direction as the z-axis, as shown in FIG. 3. Note that the calculation unit 121 only needs to be able to set the positions of point data and a virtual viewpoint based on the coordinate system of the virtual space, and may omit actually constructing a 3D point cloud model in the virtual space.

[0038] 3, 3D point cloud model B is a 3D point cloud model constructed in a virtual space by a plurality of points p1. Here, for convenience of explanation, the outline of 3D point cloud model B is shown by a solid line, but in reality it is drawn as a collection of countless points p1 arranged in the virtual space. Virtual viewpoint A is a virtual viewpoint that is the viewpoint of the operator in the virtual space, and a black dot represents the virtual viewpoint position, and an arrow extending from the black dot represents the virtual viewpoint direction.

[0039] Here, the virtual viewpoint position of virtual viewpoint A is set to the origin (0, 0, 0) in the coordinate system of the virtual space set by the calculation unit 121. In addition, the virtual viewpoint direction of virtual viewpoint A is set to the positive direction of the z axis in the coordinate system of the virtual space set by the calculation unit 121.

[0040] 4 is a diagram illustrating the interpolation processing executed by the information processing device in the embodiment. First, the distance calculation unit 122 calculates the relative distance from the virtual viewpoint A for each point p1 constituting the 3D point cloud model B. In the embodiment, the value of the relative distance of point p1 from the virtual viewpoint A is the value of the distance in the z-axis direction from the virtual viewpoint position. For example, if the coordinates of the point data of point p1 are (px, py, pz), the virtual viewpoint position in the embodiment is the origin, so the value of the relative distance of point p1 from the virtual viewpoint A is pz.

[0041] The distance calculation unit 122 determines whether each point p1 constituting the 3D point cloud model B is located within the interpolation target area based on the calculated value of the relative distance of point p1 to the virtual viewpoint A and the value of the interpolation area threshold stored in the storage unit 20 (step S2). In this embodiment, two values, L1 and L2, are set as the interpolation area threshold (L1<L2). The distance calculation unit 122 determines that a point p1 whose relative distance from the virtual viewpoint A is equal to or less than L1 is located within the first area. The distance calculation unit 122 determines that a point p1 whose relative distance from the virtual viewpoint A is greater than L1 but equal to or less than L2 is located within the second area. The distance calculation unit 122 determines that a point p1 whose relative distance from the virtual viewpoint A is greater than L2 is located within the third area (outside the interpolation target area).

[0042] Next, the interpolation data calculation unit 123 generates a number of interpolation point data based on the set value of the thinning number, based on the point data of two adjacent points that exist within the interpolation target area (step S3). In this embodiment, the interpolation data calculation unit 123 determines that two points p1 that exist on the same plane perpendicular to the virtual viewpoint direction and that are adjacent in the horizontal direction are two adjacent points p1. In other words, the interpolation data calculation unit 123 determines that two points p1 that exist on the same x-y plane perpendicular to the z-axis direction and that are adjacent in the x-axis direction are two adjacent points p1.

[0043] In this embodiment, the interpolation data calculation unit 123 generates N-1 pieces of interpolation point data between two adjacent points in the first area. Furthermore, in this embodiment, the interpolation data calculation unit 123 generates N-2 pieces of interpolation point data between two adjacent points in the second area. That is, in this embodiment, the interpolation data calculation unit 123 generates two pieces of interpolation point data for point p1 in the first area based on the point data of two adjacent points. Furthermore, the interpolation data calculation unit 123 generates one piece of interpolation point data for point p1 in the second area, which is the interpolation target area, based on the point data of two adjacent points.

[0044] After generating the interpolation point data, the interpolation data calculation unit 123 inputs the generated interpolation point data to the calculation unit 121. The calculation unit 121 arranges an interpolation point p2 in the constructed 3D point cloud model B based on the acquired interpolation point data.

[0045] The rendering unit 124 generates an image of the 3D point cloud model after the interpolation process as seen from the virtual viewpoint A based on the 3D point cloud model B set by the calculation unit 121, the viewpoint position A, and the interpolation point p2 (step S4). After executing the process of step S4, the thinned point cloud display function unit 12 ends the interpolation process. The rendering unit 124 outputs the generated image to the output unit 40, for example.

[0046] Next, effects achieved by the information processing device according to the embodiment will be described. Fig. 5 is a diagram illustrating a portion of a 3D point cloud model before interpolation as viewed from a virtual viewpoint in the example. Specifically, Fig. 5 is a diagram illustrating the field of view when point p1 on plane P at Y=0 of the 3D point cloud model B is viewed from virtual viewpoint A in Fig. 3.

[0047] 5, the closer point p1 is to the virtual viewpoint A in the depth direction (z-axis direction), the greater the apparent distance between adjacent points in the horizontal direction (x-axis direction), and the more noticeable the gaps become. Conversely, the farther point p1 is to the virtual viewpoint A in the depth direction, the smaller the apparent distance between adjacent points in the horizontal direction, and the less noticeable the gaps become. In other words, for points that are close to the virtual viewpoint A in the depth direction, the gaps between adjacent points appear to be noticeable, resulting in a decrease in apparent image quality, whereas for points that are sufficiently far from the virtual viewpoint A in the depth direction, the gaps between adjacent points appear to be narrower, and therefore, when viewed from the virtual viewpoint, no degradation in image quality occurs due to the noticeable gaps.

[0048] 6 is a diagram illustrating a portion of the interpolated 3D point cloud model as viewed from a virtual viewpoint in the embodiment, specifically, a view of point p1 on plane P at Y=0 of the 3D point cloud model B as viewed from virtual viewpoint A in FIG.

[0049] 6, for point p1 that is close to the virtual viewpoint A in the depth direction, an interpolation point p2 is placed between the point and its adjacent point in the horizontal direction, and interpolation processing is not performed for point p1 that is far from the virtual viewpoint A in the depth direction. Also, the closer a point is to the virtual viewpoint, the more interpolation points are placed between adjacent points.

[0050] This makes it possible to suppress deterioration in the apparent image quality of an image generated by the information processing device 1 according to the embodiment, in which a 3D point cloud model in a virtual space is viewed from a virtual viewpoint. Furthermore, since the information processing device 1 according to the embodiment performs interpolation processing only on point data that exists within the interpolation target area, it is not necessary to perform data processing on all point data included in the 3D point cloud data.

[0051] As described above, the information processing device 1 in the embodiment includes: a storage unit 20 that stores 3D point cloud data, which is a collection of point data including coordinate information and color information in a virtual space; a calculation unit 121 that sets a coordinate system for the virtual space and sets a virtual viewpoint including information on the position and direction of the viewpoint of an operator in the virtual space; a distance calculation unit 122 that calculates, for each point data included in the 3D point cloud data, a relative distance between the virtual viewpoint and a point in the virtual space based on the point data in the direction of the virtual viewpoint and determines whether the point data exists within an interpolation target area based on the value of the relative distance and a predetermined first threshold; an interpolation data calculation unit 123 that generates interpolated point data for each point data determined by the distance calculation unit 122 to exist within the interpolation target area based on point data of two adjacent points in the virtual space; and a rendering unit 124 that generates an image of the virtual space as seen from the virtual viewpoint using the 3D point cloud data, the virtual viewpoint, and the interpolated point data. This makes it possible to display the 3D point cloud data with a small amount of data processing while suppressing deterioration in apparent image quality when outputting the image.

[0052] In the embodiment, the 3D point cloud data stored in the storage unit 20 is thinned-out 3D point cloud data obtained by thinning out point data included in the 3D point cloud data in the point cloud data reduction unit 11 based on a preset thinning number setting value. The thinned-out 3D point cloud data has a smaller data volume than the data before thinning-out, which reduces the burden of data processing.

[0053] In the embodiment, the interpolation data calculation unit 123 determines, as two adjacent points in the virtual space, a point that exists within the interpolation target area in the virtual space and another point that exists within the interpolation target area and whose distance from the point is equal to or less than a predetermined second threshold (interpolation point distance threshold). In the interpolation process executed by the information processing device 1, there is no need to perform the interpolation process on point data that has no other points around it, so the processing load can be further reduced by performing the interpolation process only if there is a point within a distance value equal to or less than the predetermined threshold.

[0054] Next, a second embodiment of the present invention will be described below with reference to Fig. 7. Fig. 7 is a diagram showing an example of the configuration of an information processing apparatus according to the second embodiment.

[0055] Compared to the information processing device according to the embodiment, the information processing device according to the second embodiment further includes a point cloud recognition execution unit 125 that executes point cloud recognition processing as a configuration of the thinned point cloud display function unit 12 of the control unit 10.

[0056] The point cloud recognition execution unit 125 executes a process of assigning class labels to point data to be processed using a trained model trained to estimate class labels for each piece of point data included in the 3D point cloud data. A class label is a label that classifies which object each piece of point data included in the 3D point cloud data represents, such as a road, a building, a tree, or a person. The class label assignment process is a so-called semantic segmentation process, in which a class label is assigned to each piece of point data in the 3D point cloud data to be processed. The trained model may be any model that has been trained to be able to input 3D point cloud data and execute a classification process of assigning a class label to each piece of point data in the 3D point cloud data.

[0057] FIG. 8 is a diagram illustrating an example of point cloud recognition processing executed by an information processing apparatus according to the second embodiment. The upper part of FIG. 8 shows 3D point cloud data obtained by measuring a car traveling on a roadway as an example of 3D point cloud data to be processed. For ease of explanation, the contours of the objects are shown with solid lines, but it is assumed that the objects are actually measured as 3D point cloud data. The lower part of FIG. 8 shows 3D point cloud data after the point cloud recognition execution unit 125 has performed a class label assignment process on each point data in the 3D point cloud data. In FIG. 8 , in the 3D point cloud data after processing, areas classified with the same class label, such as cars, pillars, walls, trees, buildings, and ground, are illustrated with the same type of hatching. The point cloud recognition execution unit 125 classifies each point data included in the input 3D point cloud data to determine which object the point represents and assigns a class label to it.

[0058] In the second embodiment, the thinned point cloud display function unit 12 executes the following process before performing the interpolation process by the thinned point cloud display function unit 12 .

[0059] First, the point cloud recognition execution unit 125 assigns a class label to each point data included in the acquired post-thinning 3D point cloud data 22. Here, the point cloud recognition execution unit 125 performs processing such as removing noise for point clouds to which no class label is assigned.

[0060] Next, the point cloud recognition execution unit 125 classifies each piece of point data to which a class label has been assigned into point data to which a class label of an artificial object, such as a car, wall, or building, has been assigned, and point data to which a class label of a natural object, such as a tree or the ground, has been assigned. The point cloud recognition execution unit 125 notifies the distance calculation unit 122 of point data to which a class label of an artificial object has been assigned, from among the point data included in the thinned 3D point cloud data 22. The thinned point cloud display function unit 12 determines that the point data to which the class label of an artificial object has been assigned by the point cloud recognition execution unit 125 is to be subjected to interpolation processing, and executes the interpolation processing described above.

[0061] The point cloud recognition execution unit 125 determines that point data assigned a class label of a natural object, among the point data included in the thinned 3D point cloud data 22, is not subject to interpolation processing, and notifies the calculation unit 121. For the point data assigned a class label of a natural object notified by the point cloud recognition execution unit 125, the calculation unit 121 increases the diameter of points arranged in virtual space based on the point data. For example, for point data assigned a class label of a natural object, the calculation unit 121 doubles the diameter of points arranged in virtual space based on the point data by the value of the thinning number setting value.

[0062] As described above, the information processing device according to the second embodiment performs point cloud recognition as a preprocessing step prior to interpolation of the stored thinned 3D point cloud data 22. For point data determined to be part of a point cloud constituting a natural object, the device only enlarges the displayed points without performing interpolation. For man-made objects with many flat surfaces, enlarging the displayed points when modeling makes the shape difficult to discern. However, for natural objects with uneven surfaces, such as trees or rocky areas, enlarging the displayed points when modeling makes it difficult to notice changes in shape. Therefore, the information processing device according to the second embodiment can minimize degradation in the apparent image quality when outputting an image. Furthermore, simply enlarging the size (area) of points imposes a lower processing load than interpolation. Therefore, the information processing device according to the second embodiment can reduce the amount of data processing compared to the above-described embodiments.

[0063] It should be noted that the present invention is not limited to the above-described embodiment and the second embodiment. For example, the control unit 10 of the information processing device 1 can perform image processing such as a Gaussian filter on the image data generated by the rendering unit 124 as post-processing after the interpolation process. This can further improve the image quality of an image in which degradation in image quality has been suppressed by the interpolation process.

[0064] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0065] The information processing device 1 may be configured as a single server device, or may be configured as multiple server devices connected via a network. As an example, the information processing device 1 may include a server device including a point cloud data reduction unit 11, a storage unit 20 including 3D point cloud data 21 and thinned 3D point cloud data 22, and a server device including a thinned point cloud display function unit 12. Even in this case, the same effects as those of the above-described embodiment can be obtained.

[0066] DESCRIPTION OF SYMBOLS 1... Information processing device 10... Control unit 11... Point cloud data reduction unit 12... Thinned point cloud display function unit 20... Storage unit 21... 3D point cloud data 22... Thinned 3D point cloud data 30... Input unit 40... Output unit 50... Communication unit 121... Calculation unit 122... Distance calculation unit 123... Interpolation data calculation unit 124... Rendering unit 125... Point cloud recognition execution unit A... Virtual viewpoint B... 3D point cloud model P... Plane p1... Point p2... Interpolation point

Claims

1. An information processing device comprising: a storage unit that stores 3D point cloud data, which is a collection of point data including coordinate information and color information in a virtual space; a calculation unit that sets a coordinate system for the virtual space and sets a virtual viewpoint including information on the position and direction of the viewpoint of an operator in the virtual space; a distance calculation unit that calculates, for each of the point data included in the 3D point cloud data, a relative distance between the virtual viewpoint and a point in the virtual space based on the point data in the direction of the virtual viewpoint, and determines whether the point data is present within an interpolation target area based on the value of the relative distance and one or more predetermined first thresholds; an interpolation data calculation unit that generates interpolated point data for the point data determined by the distance calculation unit to be present within the interpolation target area based on the point data of two of the points adjacent to each other in the virtual space; and a rendering unit that generates an image of the virtual space as seen from the virtual viewpoint using the 3D point cloud data, the virtual viewpoint, and the interpolated point data.

2. The information processing device described in claim 1, wherein the 3D point cloud data stored in the memory unit is thinned-out 3D point cloud data after a thinning process is performed to thin out the point data contained in the 3D point cloud data based on a predetermined thinning number setting value.

3. The information processing device according to claim 2, wherein the interpolation data calculation unit generates the number of interpolation point data based on the thinning number setting value for the point data determined to be within the interpolation target area, based on the point data of two adjacent points in the virtual space.

4. An information processing device according to any one of claims 1 to 3, wherein the interpolation data calculation unit generates the interpolation point data for the point data determined to be within the interpolation target area so as to evenly arrange the interpolation points on a straight line connecting two adjacent points in the virtual space.

5. The information processing device according to claim 1, wherein the interpolation data calculation unit determines that the point located within the interpolation target area and another point located within the interpolation target area and whose distance value from the point is equal to or less than a second threshold value set in advance are two adjacent points in the virtual space.

6. The information processing device according to claim 1, wherein the interpolation data calculation unit determines that a point existing within the interpolation target area and another point existing on a plane perpendicular to the direction of the virtual viewpoint, including the point, and adjacent to the point in any direction are two adjacent points in the virtual space.

7. An information processing device as described in claim 1, further comprising a point cloud recognition execution unit that assigns class labels to the point data that classify which object each of the point data included in the 3D point cloud data represents, and determines whether or not each of the point data included in the 3D point cloud data is subject to interpolation processing based on the class labels assigned to the point data, wherein the distance calculation unit calculates the distance from the virtual viewpoint for the point data determined by the point cloud recognition execution unit to be subject to interpolation processing, and the calculation unit increases the diameter of the points in the virtual space that are placed based on the point data determined by the point cloud recognition execution unit not to be subject to interpolation processing.

8. An information processing method executed by an information processing device, comprising: setting a coordinate system for a virtual space; and setting a virtual viewpoint including information on the position and direction of the viewpoint of an operator in the virtual space; for each of the point data included in 3D point cloud data, which is a collection of point data including coordinate information and color information in the virtual space, calculating the relative distance between the virtual viewpoint and a point in the virtual space based on the point data in the direction of the virtual viewpoint; determining whether the point data exists within an interpolation target area based on the value of the relative distance and a predetermined first threshold; generating interpolated point data for the point data determined to exist within the interpolation target area based on the point data of two adjacent points in the virtual space; and generating an image of the virtual space seen from the virtual viewpoint using the 3D point cloud data, the virtual viewpoint, and the interpolated point data.

Images