Three-dimensional data processing device and three-dimensional data processing method

Abstract

To provide a technique capable of improving a compression ratio of three-dimensional data and reducing a data amount in recording and transmitting the three-dimensional data.SOLUTION: A three-dimensional data processing apparatus according to the present information detects a first region at a first time and a second region at a second time as regions including a subject in three-dimensional data, respectively, and calculates a difference in the three-dimensional data between the first time and the second time by using a coordinate transformation between the first region and the second region.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for processing three-dimensional data that describes a subject existing in three-dimensional space. [Background technology]

[0002] There are various technologies for measuring objects that exist in three-dimensional space.

[0003] Patent Document 1 addresses the issue of "compressing data volume when enabling viewing of a space or a subject located therein from different directions in video distribution," and describes the following technology (see Abstract): "Based on one or more pieces of video data received from a camera 210 capturing the 3D space to be distributed or one or more subjects located therein, a distribution device 220 recognizes one or more subjects in the 3D space as 3D objects and transmits the data to a user terminal 230. Moving 3D objects are then detected and motion data indicating the differences between them is generated. The distribution device 220 transmits the generated motion data to a user terminal 230, and the user terminal 230 moves the 3D object data stored therein based on the received motion data and displays the 3D object data on a display screen as an image from a certain line of sight along with other 3D or 2D objects that are stationary during the period corresponding to the motion data."

[0004] Patent Document 2 aims to "provide a technology for switching between intra-frame compression and inter-frame predictive differential compression for each image region," and describes the following technology (see abstract): "The image compression device of the present invention includes a motion vector detection unit, a determination unit, an intra-frame compression unit, and an inter-frame compression unit. The motion vector detection unit determines motion vectors between image data at multiple positions on the image. The determination unit determines whether the spatial change in the motion vector on the image is large. The intra-frame compression unit selects, from the image data to be compressed, image region A for which the spatial change in the motion vector is determined to be large, and performs intra-frame compression on the selected image region A. The inter-frame compression unit selects, from the image data to be compressed, image region B for which the spatial change in the motion vector is determined to be small, and performs inter-frame predictive differential compression on the selected image region B." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-033107 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-079238 Summary of the Invention [Problem to be solved by the invention]

[0006] The methods in Patent Documents 1 and 2 compress data based on the motion and motion vectors between image data. However, when handling three-dimensional data, the data volume is still large, which puts strain on the communication bandwidth during data transmission, so further reduction in data volume is required. The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology that makes it possible to improve the compression rate of three-dimensional data and reduce the amount of data when recording and transmitting three-dimensional data. [Means for solving the problem]

[0007] The three-dimensional data processing device of the present invention detects a first area at a first time and a second area at a second time as areas containing a subject within three-dimensional data, and calculates the difference in the three-dimensional data between the first time and the second time using coordinate transformation between the first area and the second area. [Effects of the Invention]

[0008] The three-dimensional data processing device according to the present invention can improve the compression rate of three-dimensional data and reduce the amount of data when recording and transmitting the three-dimensional data. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of an installation environment of a three-dimensional data processing device 1 according to a first embodiment. [Figure 2] 1 is a functional block diagram of a three-dimensional data processing device 1. FIG. [Figure 3] 10 is a diagram showing an example of a cluster managed by the region management unit 215, and a specific example of processing performed by the neighborhood distance calculation unit 216, the difference determination unit 217, and the three-dimensional information management unit 219. FIG. [Figure 4] An example of a processing flow performed by the three-dimensional data processing device 1 is shown. [Figure 5] FIG. 1 is a diagram showing an example of the configuration of a three-dimensional data processing device 1 according to a second embodiment. [Figure 6] FIG. 10 is a functional block diagram of a three-dimensional data processing device 1 according to a second embodiment. [Figure 7A] 10 shows an example of a processing flow performed by the three-dimensional data processing device 1 in the second embodiment. [Figure 7B] 10 shows an example of a processing flow performed by the three-dimensional data processing device 1 in the second embodiment. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of a three-dimensional data processing device 1 according to a third embodiment. [Figure 9]FIG. 10 is a functional block diagram of a three-dimensional data processing device 1 according to a third embodiment. [Figure 10A] 10 shows an example of a processing flow performed by the three-dimensional data processing device 1 in the third embodiment. [Figure 10B] 10 shows an example of a processing flow performed by the three-dimensional data processing device 1 in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings used to describe the embodiments, identical components are generally designated by the same reference numerals, and repeated description thereof will be omitted. Furthermore, in the following embodiments, the components (including element steps, etc.) are not necessarily essential unless otherwise specified or considered to be clearly essential in principle. Furthermore, when the terms "consisting of A," "made of A," "having A," or "including A" are used, it goes without saying that they do not exclude other elements, unless otherwise specified to include only that element. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., this includes those that are substantially similar or similar to the shape, etc., unless otherwise specified or considered to be clearly essential in principle.

[0011] The designations "first," "second," "third," etc. in this specification are used to identify components and do not necessarily limit the number, order, or content thereof. Furthermore, numbers used to identify components are used in different contexts, and numbers used in one context do not necessarily indicate the same configuration in another context. Furthermore, this does not prevent a component identified by a certain number from also serving the function of a component identified by another number.

[0012] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings etc.

[0013] <First Embodiment> FIG. 1 is a diagram illustrating an example of an installation environment for a three-dimensional data processing device 1 according to a first embodiment of the present invention. The three-dimensional data processing device 1 generates and compresses three-dimensional information using information acquired from a distance measuring device 102. The distance measuring device 102 has a field of view 103, captures image information including objects 104 and 105 within the field of view 103, and outputs the image information and camera parameters to the three-dimensional data processing device 1. The object 104 is an object at the time when the distance measuring device 102 captured the image at the current time. The object 105 is an object at the time when the distance measuring device 102 captured the image at a time prior to the current time.

[0014] The image information captured by the distance measuring device 102 is a distance image with distance information for each pixel. However, the device may also capture not only distance images but also RGB images in which each pixel has general grayscale information (such as RGB). Furthermore, the distance measuring device 102 is not limited to the type, arrangement, or circuit configuration of its internal sensors, as long as it can measure distance. Examples include a stereo camera with two complementary metal oxide semiconductor (CMOS) image sensors, a time-of-flight (ToF) sensor that combines an infrared laser with an image sensor, a structured light sensor that combines a projected pattern light-emitting element with an image sensor, or a sensor that combines these with a simple RGB camera and adjusts the relationship between the pixel positions. Alternatively, the device may be capable of measuring distance from only RGB images. For example, some devices learn the correspondence between RGB images and distance images in advance using machine learning or learn the relationship between RGB images and light fields using differentiable rendering, and then only use RGB images as input.

[0015] In this embodiment, a distance measuring device 102 captures an image of a subject moving in real space, and the three-dimensional data is compressed based on the relative relationship between subjects 104 and 105 captured at the past and present times. Although only one type of subject is present in Fig. 1, the processing described below includes cases where one or more subjects are captured.

[0016] 2 is a functional block diagram of the three-dimensional data processing device 1. The three-dimensional data processing device 1 includes an information acquisition unit 21 (data acquisition unit), an information management unit 22, a region information calculation unit 23 (region calculation unit), a conversion information calculation unit 24 (conversion calculation unit), and a difference information calculation unit 25 (difference calculation unit). The information acquisition unit 21 includes an image acquisition unit 201 and a three-dimensional conversion unit 202. The information management unit 22 includes a coordinate system management unit 203, a grid management unit 205, a feature amount management unit 209, a conversion information management unit 212, a region management unit 215, and a three-dimensional information management unit 219. The region information calculation unit 23 includes a grid generation unit 204, a moving object determination unit 206, and a connection relationship generation unit 207. The conversion information calculation unit 24 includes a feature amount calculation unit 208, a feature amount identification unit 210, a coordinate transformation calculation unit 211, a coordinate transformation estimation unit 213, and a region control unit 214. The difference information calculation unit 25 includes a neighborhood distance calculation unit 216 , a difference determination unit 217 , and a masking unit 218 .

[0017] The image acquisition unit 201 acquires image information and camera parameters from the distance measurement device 102 and outputs them to the three-dimensional conversion unit 202 .

[0018] The 3D conversion unit 202 converts the image information acquired from the image acquisition unit 201 into a set of 3D coordinate points (hereinafter referred to as point cloud data) by coordinate transformation using the image information and camera parameters, and outputs the converted image information to the grid management unit 205. Note that even if the image information is converted into surface data in which three or more coordinate points are spatially interpolated by surfaces, or into volume data in which four or more coordinate points are spatially interpolated by polyhedrons, without relying on point cloud data, this device can be implemented by referring to vertex coordinates, and therefore point cloud data will be used in the following explanation.

[0019] The coordinate system management unit 203 manages information about the world coordinate system that serves as a reference when handling point cloud data within the three-dimensional data processing device 1, and outputs information about the world coordinate system to the grid generation unit 204 in response to a request from the grid generation unit 204. The information to be managed includes, for example, whether the system is right-handed or left-handed, a definition of the vertical axis relative to the ground such as Z-up or Y-up, and vertex information of a bounding box that indicates the valid range of the data. Furthermore, based on information such as the position and orientation of the actually installed distance measuring device 102, it may be defined that the world coordinates and tilt angle of the distance measuring device 102 in the world coordinate system are set as the origin of the point cloud data.

[0020] The grid generation unit 204 references information about the world coordinate system defined by the coordinate system management unit 203, generates a grid by dividing space into a lattice pattern, and outputs the grid to the grid management unit 205. The grid may be generated just once at the start of processing, by equally dividing the space with rectangular parallelepipeds having a certain width, or may be generated as needed using a search tree such as an octree, in which an area containing point cloud data is divided into small grids and an empty space is divided into large grids.

[0021] The grid management unit 205 acquires point cloud data from the 3D conversion unit 202 and grid information from the grid generation unit 204. It associates index information indicating grid addresses with the point cloud data occupying each grid, calculates the density of the point cloud data occupying each grid and the position of the representative point of the point cloud data occupying each grid, and outputs the index information, point cloud data density information, and representative point position to the moving object determination unit 206. For example, the representative point position is calculated by calculating the center of gravity of the point cloud data occupying each grid. The grid management unit 205 also acquires indexes of grids determined to be moving objects and indexes of grids that have not changed for a certain period of time from the moving object determination unit 206, counts the number of frames in which a grid that has not changed for a certain period of time remains unchanged, and associates it as background when the number exceeds the certain number. The grid management unit 205 then outputs the indexes of grids determined to be moving objects, their representative points, and the point cloud data within the grid to the feature calculation unit 208. Additionally, the grid management unit 205 outputs point cloud data corresponding to the index information requested by the masking unit 218 to the masking unit 218. Hereinafter, a grid determined to be a moving object will be referred to as a moving object grid, a grid that does not change for a certain period of time will be referred to as a semi-background grid, and a grid registered as a background will be referred to as a background grid.

[0022] The moving object determination unit 206 acquires the index information, density information, and representative point positions of the point cloud from the grid management unit 205, determines whether each grid is a moving object, and if it is not a moving object, determines it as a quasi-background grid. The moving object grid index information and quasi-background grid index information are output to the grid management unit 205, and the moving object grid index information and representative point positions are output to the connection relationship generation unit 207. The determination of whether it is a moving object is made, for example, by whether the density change in the background grid exceeds a certain threshold for more than a certain period of time. The determination of whether it is a quasi-background is made, for example, by whether the density change in the moving object grid remains unchanged for more than a certain period of time.

[0023] The connection relationship generation unit 207 acquires the index information and representative point positions of the moving body grids from the moving body determination unit 206, groups adjacent moving body grids as being in the same area, assigns a uniform group index (hereinafter referred to as a cluster identification index) to each grouped moving body grid group (hereinafter referred to as a cluster), and outputs the index information of the moving body grids, the representative point positions, and the cluster identification index to the area management unit 215.

[0024] The feature calculation unit 208 acquires the moving object grid index, the representative point, and point cloud data within the grid from the grid management unit 205, and outputs the calculated feature amount to the feature management unit 209, linking it to the moving object grid index. The feature amount is calculated, for example, by calculating SHOT (Signature of Histograms of Orientations) or FPFH (Fast Point Feature Histograms) for the point cloud within a voxel centered on the representative point.

[0025] The feature amount management unit 209 acquires the indices of moving object grids linked to the feature amounts from the feature amount calculation unit 208 and manages them for a certain period of time. The feature amount management unit 209 also outputs the indices of moving object grids linked to the feature amounts for a period of time requested by the feature amount identification unit 210 to the feature amount identification unit 210. The period to be managed may be, for example, two time periods including information acquired at the current time and information acquired at a past time, or a period of several time periods in the past including the current time.

[0026] The feature amount identification unit 210 requests and acquires index information of moving object grids associated with feature amounts for a certain period including the present, which is stored in the feature amount management unit 209, requests and acquires representative point positions of the moving object grids and cluster identification indices from the region management unit 215, identifies feature amounts in each current cluster and feature amounts in each past cluster, and outputs the index information of the moving object grids in each cluster, the representative point positions, and the relationship between corresponding points from past time to the present time to the coordinate transformation calculation unit 211. The feature amount identification unit 210 also acquires the coordinate transformation calculated for each cluster from the coordinate transformation calculation unit 211, performs coordinate transformation on each representative point position at past time, and if the total distance between corresponding points at each representative point position at the present time and past time after the coordinate transformation is less than a certain threshold or if the number of iterative calculations is equal to or greater than a certain number, outputs an identification completion signal to the coordinate transformation calculation unit 211 and outputs each identified cluster identification index at the current time to the region management unit 215. Identification methods include nearest neighbor search libraries such as FLANN (Fast Library for Approximate Nearest Neighbors) and fitting methods such as RANSAC (Random Sample Consensus) and ICP (Iterative Closest Point).In addition, when there are multiple clusters with similar shapes, there is a possibility that mismatches will occur between clusters that are not spatially close.In such cases, cost minimization can be performed using the Hungarian method, which uses the distance between each representative point position as a cost, to match nearby clusters and prevent mismatches.

[0027] The coordinate transformation calculation unit 211 acquires the index information and representative point positions of the moving body grid in each cluster, and the relationship between corresponding points from the past time to the present time, from the feature amount identification unit 210, and calculates the coordinate transformation between each corresponding point. Furthermore, when it acquires a signal indicating completion of identification from the feature amount identification unit 210, it outputs the current coordinate transformation to the transformation information management unit 212. For example, in the case of rigid body transformation, the coordinate transformation is calculated using Equation 1, where X is the set of representative point positions at the past time, Y is the set of representative point positions at the present time, R is the rotation matrix, and t is the translation vector. The initial values ​​of R and t may be a unit matrix and a zero vector, respectively, or an existing coordinate transformation managed by the transformation information management unit 212 may be used as the initial values.

[0028]

number

[0029] The transformation information management unit 212 acquires coordinate transformations for each cluster from the coordinate transformation calculation unit 211 and the coordinate transformation estimation unit 213, calculates a state space based on the coordinate transformations for several time periods it manages, and manages the coordinate transformations and state space for a certain period. It also outputs the state space of each cluster to the coordinate transformation estimation unit 213 in response to a request from the coordinate transformation estimation unit 213. It also outputs the coordinate transformation of each cluster to the coordinate transformation calculation unit 211 and the masking unit 218 in response to a request from the coordinate transformation calculation unit 211 and the masking unit 218. The managed period includes several time periods in the past, including the current time, and several time periods in the future estimated by the coordinate transformation estimation unit. When a newly calculated coordinate transformation is acquired as time progresses, it updates the estimated coordinate transformation of the state, and updates the state space accordingly. Alternatively, it is possible to manage these using a graph structure, with movement coordinate points and posture changes from the start time as nodes and relative coordinate transformations between nodes as edges, and to impose constraints related to the measurement environment and loop closure.

[0030] The coordinate transformation estimation unit 213 acquires the state space for each cluster from the transformation information management unit 212, estimates the coordinate transformation several time periods into the future, and outputs the estimated coordinate transformation to the transformation information management unit 212 and the region control unit 214. Methods for estimating the coordinate transformation include, for example, a Kalman filter, a particle filter, or a Bayesian filter.

[0031] The region control unit 214 acquires the coordinate transformation estimated by the coordinate transformation estimation unit 213 for several time periods in the future, and outputs control information to the region management unit 215 in accordance with the coordinate transformation. For example, if the coordinate transformation is a rigid transformation matrix, the control information is the rigid transformation matrix. Also, a scaling value such as expanding the region by 5% may be used as the control information, or the processing speed may be monitored internally and the scaling value may be adjusted according to the processing speed.

[0032] The region management unit 215 acquires moving object grid index information, representative point positions, and cluster identification indices from the connection relationship generation unit 207 and manages them for a certain period of time. It also acquires and updates each cluster identification index identified from the feature quantity identification unit 210. Meanwhile, separately from the information observed at the current time, i.e., the information acquired from the connection relationship generation unit 207, it manages information on each cluster estimated for several time periods in the future (hereinafter referred to as estimated clusters). The estimated clusters are obtained by controlling each cluster at the current time in accordance with control information acquired from the region control unit 214. Examples of control processing include performing coordinate transformation on each representative point to update the position, orientation, or shape, or a combination of these pieces of information, isotropically expanding or contracting from the center of gravity of each representative point, and rotating or deforming based on the center of gravity of each representative point. When expanding or contracting, the indexes of the surrounding grids for each grouped moving object grid are referenced, and an index is added or deleted from each cluster. Furthermore, the index information of the moving object grid, the representative point positions, and the index of each group for the period requested by the neighborhood distance calculation unit 216 are output to the neighborhood distance calculation unit 216 .

[0033] The neighborhood distance calculation unit 216 acquires the index information, representative point position, and cluster identification index of the moving object grid at the current time from the region management unit 215, acquires the index information, representative point position, and cluster identification index of the moving object grid of the estimated cluster, searches for neighboring points between each other using the representative point positions of each cluster, calculates the distance between them, and outputs the representative point positions of each cluster and each estimated cluster at the current time, as well as the index information of each cluster linked to the calculated distance, to the difference determination unit 217. The search for neighboring points can be performed quickly, for example, by determining whether a representative point position of the estimated cluster is located at or near the same position as a representative point position of the cluster at the current time. If there is no representative point position, the distance to the grid index is set to infinity; if there is, the distance between the representative points is calculated. If the representative point position of the cluster at the current time is not located near the representative point position of the estimated cluster, the distance to the grid index is set to a negative value. Furthermore, in the initial search, the estimated cluster is set to null information, i.e., the distance to all indices of the moving object grid at the current time is set to infinity. The significance of setting the distance to infinity will be discussed later.

[0034] The difference determination unit 217 obtains the representative point positions, grid indices, and cluster identification indexes of each cluster and each estimated cluster at the current time linked with the distance information from the neighborhood distance calculation unit 216, determines that there is a difference if the respective distances are equal to or greater than a certain threshold, determines that there is no difference if the opposite is true, and determines that there is an inverse difference if the opposite is true, and outputs the indices of the grid of each cluster together with the cluster identification index to the masking unit 218. The significance of the difference and inverse difference will be described later.

[0035] The masking unit 218 obtains the representative point positions and grid indices of each cluster associated with a difference flag, and the cluster identification index from the difference determination unit 217, obtains point cloud data corresponding to the grid indices of each cluster with a difference flag from the grid management unit 205, obtains coordinate transformations corresponding to each cluster identification index from the transformation information management unit 212, and outputs the point cloud data and coordinate transformations, as well as index information of the representative point positions and grids with a no-difference flag, and the representative points and grid indices with an inverse difference flag to the three-dimensional information management unit 219.

[0036] The 3D information management unit 219 acquires and manages the point cloud data and coordinate transformation for each cluster, the representative point positions and grid indices flagged with no difference, and the representative point positions and grid indices flagged with inverse difference from the masking unit 218. When restoring the point cloud data at the current time, the point cloud data portion is left as is, and for the grid indices flagged with no difference, an inverse coordinate transformation is performed on the representative point positions, and the point cloud data occupying the corresponding grid indices at the past time is copied and coordinate transformation is performed again. Note that the copy operation is performed on all grid indices other than those flagged with inverse difference. Specific examples of inverse coordinate transformation and the like will be described later.

[0037] 3 is a diagram showing an example of a cluster managed by the region management unit 215, and a specific example of the processing performed by the neighborhood distance calculation unit 216, the difference determination unit 217, and the three-dimensional information management unit 219. The processing progresses from step 31 to step 34.

[0038] Step 31 is a cluster 301 of subject 104 at the current time and a cluster 302 of subject 105 at the past time, which are managed by region management unit 215 in a situation where subject 105 at the past time has moved to the position of subject 104 at the current time. Cluster 303 is an estimated cluster at the past time, and is obtained by expanding the region of cluster 302. The estimated cluster is a region estimated at the past time from a region assumed to include subject 104 at the current time. In order to reliably include subject 104, it is desirable to expand cluster 302 (a group of grids in which subject 105 existed at the past time). Therefore, estimated cluster 303 is generated by expanding cluster 302.

[0039] Step 32 is an example of the result of processing performed by the neighborhood distance calculation unit 216 in step 31, and is the calculation of the nearest neighbor distance between each representative point position of the estimated cluster 303 and each representative point position of the cluster 301.

[0040] Neighborhood distance 304 is an area where there is no representative point position of estimated cluster 303 in the same grid as the representative point position of cluster 301, and therefore the distance is infinite. Cluster 301 represents the current position and current shape of subject 104. The cluster corresponding to neighborhood distance 304 means that the current position and current shape of subject 104 deviate from estimated cluster 303 estimated at a past time. For example, neighborhood distance 304 occurs when the subject deforms. To represent such a deviating shape, neighborhood distance 304 is set to infinity. It does not necessarily have to be an infinite distance as long as it can express the same thing.

[0041] Neighborhood distance 305 is an area where the representative point position of estimated cluster 303 is in the same grid as the representative point position of cluster 301, and the distance between the two points is equal to or greater than a threshold. The cluster corresponding to neighborhood distance 305 is within the range of estimated cluster 303, which was estimated at a past time for the current position and current shape of subject 104, but the distance is relatively large compared to neighborhood distance 306. For example, the part where the deformation of the subject is contained within estimated cluster 303 is neighborhood distance 305.

[0042] Neighborhood distance 306 is an area where the representative point position of estimated cluster 303 is in the same grid as the representative point position of cluster 301, and the distance between the two points is less than a threshold value. The cluster corresponding to neighborhood distance 306 is an area where the subject is contained within estimated cluster 303 and where the shape is not deformed (or is slightly deformed).

[0043] Neighborhood distance 307 is an area where the distance between the representative point position of estimated cluster 303 and the representative point position of cluster 301 is a negative value because there is no representative point position of cluster 301 in the same grid. Since estimated cluster 303 is an expanded version of cluster 302, there are parts of estimated cluster 303 where subject 104 does not exist. To represent this, a negative value is set for neighborhood distance 307. It does not necessarily have to be a negative distance as long as it can represent the same thing.

[0044] Step 33 is an example of the results of processing performed by the difference determination unit 217 in step 32, where a difference is determined from the distance associated with each grid. Difference 308 is a grid determined to have a difference because neighborhood distances 304 and 305 are equal to or greater than a threshold. Difference 309 is a grid determined to have no difference because neighborhood distance 306 is less than a threshold. Difference 310 is a grid determined to have an inverse difference because neighborhood distance 307 is a negative value.

[0045] Step 34 is an example of processing performed by the 3D information management unit 219, and restores the point cloud data at each managed time. The moving body grid group 311 is the grid corresponding to the past time when inverse coordinate transformation is performed on the representative point position of the difference 309. If the point cloud data belonging to this moving body grid group 311 is again coordinate transformed and copied to the corresponding grid at the current time, and then integrated with the difference 308, the original point cloud data can be restored.

[0046] The reason for re-coordinate-transforming the inversely transformed grid in step 34 will now be explained. If the three-dimensional data at the current time is to be presented to the user immediately, it is sufficient to present the three-dimensional data of cluster 301. On the other hand, if the three-dimensional data at the current time is to be presented to the user later, the three-dimensional data at the current time must be temporarily saved and then restored later. By saving only the coordinate transformation of grids that have not changed (or have changed little), the amount of data stored can be reduced. However, in this case, since the only three-dimensional data saved is that from the past (cluster 302), an inverse transformation is first performed toward the past, and the three-dimensional data from the past is obtained and then re-coordinate-transformed. This makes it possible to accurately restore unchanged parts while reducing the amount of data stored.

[0047] FIG. 4 shows an example of a processing flow performed by the three-dimensional data processing device 1. In step S401, the grid generation unit 204 generates a grid based on information from the coordinate system management unit 203. In step S402, the image acquisition unit 201 acquires image information from the distance measurement device 102. In step S403, the three-dimensional conversion unit 202 converts the image information into point cloud data. In step S404, the moving object determination unit 206 determines whether the moving object grid is a moving object grid or a quasi-background grid. In step S405, the connection relationship generation unit 207 groups the moving object grids to generate clusters. In step S406, the feature calculation unit 208 calculates the feature of the moving object grid. In step S407, if the current processing is the first time, step S413 is performed; if not, step S408 is performed. In step S408, the feature identification unit 210 identifies the feature of each cluster at the past time and the feature of each cluster at the current time, and calculates the total distance between each representative point position. In step S409, if the total distance between the representative point positions calculated by the feature quantity identification unit 210 is equal to or greater than a threshold value or if the number of iterative calculations by the feature quantity identification unit 210 and the coordinate transformation calculation unit 211 is equal to or greater than the iteration threshold value, step S411 is performed. Otherwise, step S410 is performed. In step S410, the coordinate transformation calculation unit 211 calculates coordinate transformation between each corresponding point identified by the feature quantity identification unit 210. In step S411, the coordinate transformation estimation unit 213 acquires information about the state space of each cluster and estimates coordinate transformation several time periods into the future. In step S412, the region control unit 214 generates control information for each cluster managed by the region management unit 215 based on the information about the coordinate transformation estimated for the cluster, and the region management unit 215 generates an estimated cluster based on that information. In step S413, the neighborhood distance calculation unit 216 calculates each distance between the cluster at the current time and the estimated cluster. In step S414, the difference determination unit 217 determines the difference based on the information on each distance calculated by the neighborhood distance calculation unit 216, and generates a difference flag.In step S415, based on the difference flag, the masking unit 18 outputs the point cloud data, information related to coordinate transformation, and the indexes of grids flagged as having no difference and the indexes of grids flagged as having inverse difference to the three-dimensional information management unit 219. In step S416, if a signal indicating the end of processing is input from outside, the processing ends; otherwise, the process returns to step S402 and continues.

[0048] In the first embodiment, active compression processing is described in which information about a subject several time points in the future is estimated, clusters are controlled according to the estimated information, and a difference between the estimated information and the information about the same subject actually observed at the next time point is determined. However, passive compression processing using only actually observed information is also possible. For example, the coordinate transformation estimation unit 213 does not estimate coordinate transformation several time points in the future, but instead acquires information about the coordinate transformation calculated by the coordinate transformation calculation unit 211 from the transformation information management unit 212 and outputs the information directly to the region control unit 214 as estimated information. The region control unit 214 also outputs the acquired information about the coordinate transformation directly to the region management unit 215 as control information. Therefore, the estimated clusters managed by the region management unit 215 are the same as those obtained by controlling the clusters at past times using the coordinate transformation calculated by the coordinate transformation calculation unit 211. This enables compression processing using only observation information.

[0049] <First embodiment: Summary> According to the three-dimensional data processing device 1 of the first embodiment, the change in grid unit for each moving object is extracted as a difference, and only the coordinate transformation formula for grids with small changes is saved, and new three-dimensional data is acquired for grids with large changes, thereby enabling the three-dimensional data to be compressed efficiently.

[0050] <Embodiment 2> The second embodiment of the present invention is basically the same as the first embodiment, and the following description will focus on the differences. The main difference between the second embodiment and the first embodiment is that point cloud data is transmitted. When point cloud data is stored in local storage, the configuration of the first embodiment allows compression and decompression processing to be performed by a single device. However, when decompressing data remotely, the configuration of the second embodiment allows three-dimensional data to be compressed during transmission and decompressed on the receiving side.

[0051] 5 is a diagram showing an example of the configuration of a three-dimensional data processing device 1 according to embodiment 2. In addition to the configuration described in embodiment 1, a transmission unit 501 and a reception unit 502 are newly provided. The transmission unit 501 transmits information acquired from the three-dimensional data processing device 1 to the reception unit 502. The reception unit 502 receives the information transmitted from the transmission unit 501 and outputs it to the three-dimensional information management unit 219. The transmission unit 501 and the reception unit 502 may be configured as part of the three-dimensional data processing device 1, or may be configured as separate devices.

[0052] 6 is a functional block diagram of the three-dimensional data processing device 1 according to embodiment 2. The transmission unit 501 includes a transmission information generation unit 601 and a transmission unit 602, and the reception unit 502 includes a reception unit 603 and a received information decoding unit 604. The three-dimensional information management unit 219 has been transferred from the information management unit 22 to the reception unit 502.

[0053] The masking unit 218 outputs, instead of the three-dimensional information management unit 219, to the transmission information generation unit 601, information regarding the point cloud data and coordinate transformation, the representative point positions and grid indexes with no difference flags, and the representative point positions and grid indexes with inverse difference flags.

[0054] The transmission information generation unit 601 acquires from the masking unit 218 the point cloud data and information relating to coordinate transformation in each cluster, the representative point positions and grid indices flagged with no difference, and the representative point positions and grid indices flagged with inverse difference, performs data shaping for transmission, and outputs the data to the transmission unit 602. Data shaping is performed, for example, by serializing various types of data according to a defined format, performing lossless compression using a ZIP algorithm or the like, and adding the data to the end of header information.

[0055] The transmitter 602 transmits the data acquired from the transmission information generator 601 to the receiver 603. The transmission protocol may be, for example, UDP, TCP, or WebSocket. If there is no destination, an error is output according to the timeout setting or the like, or a reconnection is periodically attempted.

[0056] The receiving unit 603 receives data from the transmitting unit 602. The received information decoding unit 604 converts the data acquired from the receiving unit 603 back into the original data string, and outputs the point cloud data and coordinate transformation information for each cluster, as well as the representative point positions and grid indices marked with no difference flags and the representative point positions and grid indices marked with inverse difference flags, to the three-dimensional information management unit 219. The conversion back to the original data string is accomplished, for example, by deserializing and decompressing the ZIP-compressed data serialized by the transmission information generating unit 601 in accordance with the header information.

[0057] 7A and 7B show an example of a processing flow performed by the three-dimensional data processing device 1 in embodiment 2. Steps S701 to S707 are newly added to FIG. 4. Steps S701 and S702 are performed on the transmitting side. Steps S703 to S707 are performed on the receiving side.

[0058] In step S701, the transmission information generation unit 601 performs data shaping on the point cloud data and coordinate transformation information for each cluster acquired from the masking unit 218, as well as the representative point positions and grid indexes flagged with no difference and the representative point positions and grid indexes flagged with inverse difference. In step S702, the transmission unit 602 transmits the data acquired from the transmission information generation unit 601.

[0059] In step S703, if a sender exists, step S704 is performed; if a sender does not exist, the process proceeds to step S704. In step S704, if a processing end signal is input from outside, the process ends; otherwise, the process returns to step S703 and continues. In step S705, the receiving unit 603 receives data from the transmitting unit 602. In step S706, the received information decoding unit 604 converts the data acquired from the receiving unit 603 into the original data string. In step S707, if a processing end signal is input from outside, the process ends; otherwise, the process returns to step S705 and continues.

[0060] <Embodiment 2: Summary> According to the three-dimensional data processing device 1 of the second embodiment, the three-dimensional data can be efficiently compressed as in the first embodiment, and then the compressed three-dimensional data can be transmitted to a receiving unit 502 (for example, a receiving device configured separately from the three-dimensional data processing device 1). The receiving unit 502 can also restore the compressed three-dimensional data as in the first embodiment. In other words, the three-dimensional data can be efficiently transmitted to a remote site.

[0061] <Third Embodiment> The third embodiment of the present invention is basically the same as the first embodiment, and the following description will focus on the differences. The third embodiment differs from the first embodiment in that point cloud data is reproduced in a virtual space, and attention is paid to moving objects in the area viewed by the user. For example, in a situation where multiple moving objects exist, if there is a moving object that is unnecessary for actual use, the configuration of the third embodiment makes it possible to select the moving object, thereby enabling further data compression.

[0062] FIG. 8 is a diagram showing an example of the configuration of a three-dimensional data processing device 1 according to the third embodiment. Compared to the configuration of the first embodiment, the third embodiment newly includes a viewpoint video generation unit 803 and a user input unit 804. Moving subjects 801 and 802 are captured by a distance measuring device 102, point cloud data is output from the three-dimensional data processing device 1 to the viewpoint video generation unit 803, and a user viewpoint video for the point cloud data in a virtual space 805 is generated in accordance with input to the user input unit 804 (field of view specification unit). Subject data 806 and 807 are point cloud data of the subjects 801 and 802 reproduced in the virtual space 805. A virtual viewpoint 808 is a user viewpoint in the virtual space 805. A field of view 809 is a field of view held by the virtual viewpoint 808. The viewpoint video generation unit 803 and the user input unit 804 may be configured as part of the three-dimensional data processing device 1 or as separate devices.

[0063] 9 is a functional block diagram of the three-dimensional data processing device 1 according to the third embodiment. The user input unit 804 includes a user input acquisition unit 901 and a user coordinate control unit 902. The virtual space management unit 903 and viewpoint video generation unit 803 include a virtual visual field generation unit 904 and a rendering unit 905. The moving object determination unit 206 and the three-dimensional information management unit 219 have been changed to a moving object determination unit 906 and a three-dimensional information management unit 907. The moving object determination unit 906 and the three-dimensional information management unit 907 are basically the same as the moving object determination unit 206 and the three-dimensional information management unit 219, so only the differences will be described below.

[0064] A user input acquisition unit 901 acquires an input signal to a user device or the like, and outputs it to a user coordinate control unit 902. Input operations include, for example, pressing a keyboard, moving or dragging a mouse, and moving or tilting the head of an HMD (Head Mounted Display).

[0065] The user coordinate control unit 902 acquires an input signal from the user input acquisition unit 901 , converts it into the amount of movement and rotational movement of the virtual viewpoint 808 , and outputs it to the virtual visual field generation unit 904 .

[0066] The virtual space management unit 903 manages spatial information, position and orientation information of the user viewpoint 808, and camera model parameters required to generate an image at the virtual viewpoint 808, outputs the spatial information, position and orientation information of the user viewpoint 808, and camera model parameters in response to a request from the virtual view field generation unit 904, and outputs spatial information in response to a request from the rendering unit 905. Furthermore, when position and orientation information of the user viewpoint 808 is input from the virtual view field generation unit 904, the virtual space management unit 903 updates the information to that information. The managed spatial information includes, for example, the origin position of the world coordinate system, whether it is a right-handed or left-handed system, a definition of a vertical axis relative to the ground such as Z-up or Y-up, and vertex information of a bounding box that indicates the range of the space, and is basically the same as the information managed by the coordinate system management unit 203.

[0067] The virtual view field generation unit 904 requests and acquires space information, position and orientation information of the user viewpoint 808, and camera model parameters managed by the virtual space management unit 903, updates the position and orientation information of the user viewpoint 808 using the movement amount and rotational movement amount of the virtual viewpoint 808 acquired from the user coordinate control unit 902, and outputs the information to the virtual space management unit 903. It also calculates a projective transformation using the position and orientation information of the user viewpoint 808 and the camera model parameters, and outputs the transformation to the rendering unit 905. It also calculates frustum information representing the field of view using the position and orientation information of the user viewpoint 808 and the camera model parameters, and outputs the information to the moving object determination unit 206. The initial values ​​of the position and orientation information of the user viewpoint 808 are, for example, the origin of the space.

[0068] The rendering unit 905 requests and acquires spatial information managed by the virtual space management unit 903, requests and acquires point cloud data managed by the three-dimensional information management unit 907, and performs rendering using the projective transformation acquired from the virtual view field generation unit 904, and displays the rendering to the user.

[0069] In addition to the processing of the moving object determining unit 206, the moving object determining unit 906 uses the frustum information acquired from the virtual visual field generating unit 904 to determine that moving object grids that do not exist within the range of the frustum are quasi-background grids.

[0070] In addition to the processing of the three-dimensional information management unit 219 , the three-dimensional information management unit 907 restores point cloud data in response to a request from the rendering unit 905 and outputs the restored data to the rendering unit 905 .

[0071] 10A and 10B show an example of a processing flow performed by the three-dimensional data processing device 1 in embodiment 3. Steps S1001 to S1008 are newly added to the flowchart in Fig. 4. Steps S1001 to S1006 are processes performed by the viewpoint video generation unit 803 and the user input unit 804.

[0072] In step S1001, the rendering unit 905 requests point cloud data from the 3D information management unit 907; if no point cloud data is available, the processing proceeds to step S1002; if no point cloud data is available, the processing proceeds to step S1003. In step S1002, if a processing end signal is input from an external device, the processing ends; otherwise, the processing returns to step S1001 and continues. In step S1003, if there is no user input signal, the user input acquisition unit 901 performs the processing of step S1005; if there is a user input signal, the processing proceeds to step S1004. In step S1004, the virtual view generation unit 904 updates the position and orientation information of the user viewpoint 808 and calculates projective transformation and frustum information. In step S1005, the rendering unit 905 renders the point cloud data. In step S1006, if there is a processing end signal input from an external device, the processing ends; otherwise, the processing returns to step S1003 and continues.

[0073] In step S1007, if there is a request from the rendering unit 905 to the three-dimensional information management unit 907, the processing of step S1008 is carried out. In step S1008, the three-dimensional information management unit 907 restores the point cloud data in response to the request from the rendering unit 905.

[0074] <Embodiment 3: Summary> According to the three-dimensional data processing device 1 of the third embodiment, in addition to the first embodiment, it is possible to select a moving object to be processed based on the user's field of view, thereby enabling further data compression.

[0075] <Modifications of the present invention> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0076] In the above embodiments, each functional unit provided in the three-dimensional data processing device 1 can be configured by hardware such as a circuit device that implements that function, or can be configured by a computing device such as a CPU (Central Processing Unit) executing software that implements that function. [Explanation of symbols]

[0077] 1...3D data processing device, 101...3D information compression unit, 102...range measuring device, 103...field of view, 104...subject, 105...subject, 21...information acquisition unit, 22...information management unit, 23...region information calculation unit, 24...conversion information calculation unit, 25...difference information calculation unit, 201...image acquisition unit, 202...3D conversion unit, 203...coordinate system management unit, 204...grid generation unit, 205...grid management unit , 206... moving object determination unit, 207... connection relationship generation unit, 208... feature amount calculation unit, 209... feature amount management unit, 210... feature amount identification unit, 212... conversion information management unit, 213... coordinate conversion estimation unit, 214... area control unit, 215... area management unit, 216... neighborhood distance calculation unit, 217... difference determination unit, 218... masking unit, 31... process, 32... process, 33... process, 34... process, 301... class data, 302...cluster, 303...estimated cluster, 304...neighborhood distance, 305...neighborhood distance, 306...neighborhood distance, 307...neighborhood distance, 308...difference, 309...difference, 310...difference, 311...moving object grid group, 501...3D information transmission unit, 502...3D information reception unit, 601...transmission information generation unit, 602...transmission unit, 603...reception unit, 604...received information decoding unit, 801...subject , 802...Subject, 803...Viewpoint video generation unit, 804...User input unit, 805...Virtual space, 806...Subject data, 807...Subject data, 808...Virtual viewpoint, 809...Virtual field of view, 901...User input acquisition unit, 902...User coordinate control unit, 903...Virtual space management unit, 904...Virtual field of view generation unit, 905...Rendering unit, 906...Moving object determination unit, 907...3D information management unit

Claims

1. A three-dimensional data processing device that processes three-dimensional data describing an object existing in a three-dimensional space, a data acquisition unit for acquiring the three-dimensional data; an area calculation unit that detects an area in which the subject exists in the three-dimensional data; a transformation calculation unit that calculates coordinate transformation of the subject between different times; a difference calculation unit that calculates a difference between the three-dimensional data between different times; Equipped with the region calculation unit detects the subject included in the three-dimensional data at a first time as a first region; the region calculation unit detects the subject included in the three-dimensional data at a second time as a second region; the transformation calculation unit calculates the coordinate transformation between the first region and the second region; the difference calculation unit calculates the difference between the first time and the second time by comparing corresponding parts of the first region and the second region using the coordinate transformation; the transformation calculation unit estimates an estimated area estimated to include the subject at the second time from the first area based on the coordinate transformation, and estimates the three-dimensional data in the estimated area; the difference calculation unit calculates the difference by comparing the three-dimensional data in the estimation region with the three-dimensional data in the second region; the region calculation unit generates a grid by dividing the three-dimensional space into a grid pattern; the difference calculation unit calculates a distance between the three-dimensional data in the second region and the three-dimensional data in the estimation region for each grid; the difference calculation unit identifies, as a difference-free grid, a grid in the second region that is included in the estimation region and whose distance is less than a threshold value; The difference calculation unit excludes the three-dimensional data in the no-difference grid from the difference. A three-dimensional data processing device.

2. the transformation calculation unit calculates the coordinate transformation by identifying at least one of a movement amount, a rotation amount, and a deformation amount of the subject between the first area and the second area; The transformation calculation unit calculates the coordinate transformation by identifying corresponding portions of the first region and the second region.

2. The three-dimensional data processing device according to claim 1.

3. The difference calculation unit calculates the three-dimensional data in a portion of the second region that is not included in the estimated region as the difference.

2. The three-dimensional data processing device according to claim 1.

4. the region calculation unit generates a grid by dividing the three-dimensional space into a grid pattern; the difference calculation unit calculates a distance between the three-dimensional data in the second region and the three-dimensional data in the estimation region for each grid; The difference calculation unit calculates, as the difference, the three-dimensional data in the grid of the second region that is included in the estimated region and whose distance is equal to or greater than a threshold.

2. The three-dimensional data processing device according to claim 1.

5. The difference calculation unit specifies a portion that is included in the estimation region and does not include the second region as the difference-free grid.

2. The three-dimensional data processing device according to claim 1.

6. the three-dimensional data processing device further comprises a data management unit that manages the three-dimensional data; The data management unit manages data describing the contents of the coordinate transformation instead of the three-dimensional data itself for the portion of the three-dimensional data corresponding to the difference-free grid, thereby reducing the amount of data to be managed.

2. The three-dimensional data processing device according to claim 1.

7. the three-dimensional data processing device further comprises a data management unit that manages the three-dimensional data; the data management unit applies an inverse transformation of the coordinate transformation to the three-dimensional data in the difference-free grid, thereby identifying the grid corresponding to the difference-free grid in the first region; the data management unit applies the coordinate transformation to the three-dimensional data belonging to the grid identified by the inverse transformation, thereby restoring the three-dimensional data belonging to the difference-free grid in the second region; The data management unit reduces the amount of data to be managed by performing the restoration for the differential-free grid instead of managing the three-dimensional data itself.

2. The three-dimensional data processing device according to claim 1.

8. the three-dimensional data processing device further includes a transmission unit that transmits the three-dimensional data; The transmission unit transmits data describing the contents of the coordinate transformation instead of the three-dimensional data itself for a portion of the three-dimensional data corresponding to the difference-free grid, thereby reducing the amount of data to be transmitted.

2. The three-dimensional data processing device according to claim 1.

9. the three-dimensional data processing device further includes a transmission unit that transmits the three-dimensional data; The transmission unit reduces the amount of data to be transmitted by transmitting information specifying the difference-free grid instead of the three-dimensional data itself for the difference-free grid.

2. The three-dimensional data processing device according to claim 1.

10. the three-dimensional data processing device further comprises a virtual space management unit that renders the three-dimensional data in a virtual space; the three-dimensional data processing device further includes a visual field specifying unit that specifies a visual field of a user viewing the rendered three-dimensional data, The area calculation unit, the transformation calculation unit, and the difference calculation unit use only the portion of the three-dimensional data included within the field of view as a processing target.

2. The three-dimensional data processing device according to claim 1.

11. the visual field specification unit receives a user input specifying a virtual visual field of the user in the virtual space; The visual field specifying unit specifies the virtual visual field specified by the user input as the visual field.

11. The three-dimensional data processing device according to claim 10.

12. A three-dimensional data processing method for processing three-dimensional data describing an object existing in a three-dimensional space, comprising: acquiring the three-dimensional data; detecting an area in the three-dimensional data where the subject exists; Calculating coordinate transformation of the object between different times; calculating a difference between the three-dimensional data at different times; and In the step of detecting the region, the subject included in the three-dimensional data at a first time is detected as a first region; In the step of detecting the region, the subject included in the three-dimensional data at a second time is detected as a second region; In the step of calculating the coordinate transformation, the coordinate transformation between the first region and the second region is calculated, In the step of calculating the difference, the difference between the first time and the second time is calculated by comparing corresponding portions of the first region and the second region using the coordinate transformation; In the step of calculating the coordinate transformation, an estimated region estimated to include the subject at the second time point is estimated from the first region based on the coordinate transformation, and the three-dimensional data in the estimated region is estimated; In the step of calculating the difference, the difference is calculated by comparing the three-dimensional data in the estimation region with the three-dimensional data in the second region; In the step of calculating the region, a grid is generated by dividing the three-dimensional space into a grid pattern; In the step of calculating the difference, a distance between the three-dimensional data in the second region and the three-dimensional data in the estimation region is calculated for each grid; In the step of calculating the difference, the grid in the second region that is included in the estimation region and whose distance is less than a threshold is identified as a difference-free grid; In the step of calculating the difference, the three-dimensional data in the difference-free grid is excluded from the difference. A three-dimensional data processing method comprising:

Images