Information processing system, information processing device, and information processing method

Abstract

The present invention suppresses degradation in the quality of a virtual space that is reproduced on the basis of point group data after data volume reduction. An information processing system according to the present invention comprises a partitioning unit that partitions a three-dimensional region that is within a virtual space and includes point group data into a plurality of partial regions and an extraction unit that extracts second point group data from first point group data that is included in the point group data of each of the partial regions in accordance with evaluation results for the shape of the first point group data.

Description

Information processing system, information processing device, and information processing method

[0001] The present disclosure relates to an information processing system, an information processing device, and an information processing method.

[0002] The amount of point cloud data that can be placed in a virtual space can be enormous, and research is being conducted to reduce the amount of data. Patent Document 1 describes a technology that determines the priority of the point cloud data in response to a display request based on the viewpoint or line of sight of a user viewing a virtual space reproduced from the point cloud data, and thins out some of the parts with lower priority.

[0003] Japanese Patent Application Publication No. 2020-136882

[0004] The technology described in Patent Document 1 has a problem in that the virtual space reproduced from the point cloud data after thinning may have reduced quality in areas with low priority.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technology for suppressing degradation in the quality of a virtual space reproduced based on point cloud data after data volume reduction.

[0006] An information processing system according to an exemplary aspect of the present disclosure includes a division means for dividing a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas, and an extraction means for extracting second point cloud data from the first point cloud data in accordance with an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0007] An information processing device according to an exemplary aspect of the present disclosure includes a division means for dividing a three-dimensional area including point cloud data in a virtual space into a plurality of partial areas, and an extraction means for extracting second point cloud data from the first point cloud data in accordance with an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0008] An information processing method according to an exemplary aspect of the present disclosure includes a division process in which at least one processor divides a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas, and an extraction process in which the at least one processor extracts second point cloud data from the first point cloud data in accordance with an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0009] According to one exemplary aspect of the present disclosure, it is possible to provide a technology that suppresses degradation in the quality of a virtual space reproduced based on point cloud data after data volume reduction.

[0010] FIG. 1 is a block diagram showing a configuration of an information processing system according to the present disclosure. FIG. 2 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 3 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 4 is a block diagram showing a configuration of an information processing system according to the present disclosure. FIG. 5 is a diagram showing an example of a distribution of an inner product of normals in a partial region according to the present disclosure. FIG. 6 is a diagram showing an example of relationship information showing the relationship between mesh quality and extraction rate according to the present disclosure. FIG. 7 is a diagram showing an example of a method for determining an extraction rate according to the present disclosure. FIG. 8 is a diagram showing an example of a method for determining an extraction rate according to the present disclosure. FIG. 9 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 10 is a diagram explaining the shape indicated by point cloud data after data volume reduction according to the present disclosure in comparison with a comparative example. FIG. 11 is a diagram showing an example of measurement results of extraction rate and CD value in data volume reduction according to the present disclosure. FIG. 12 is a diagram showing a functional block configuration of an information processing system according to the present disclosure. FIG. 13 is a block diagram showing the hardware configuration of a computer functioning as each device according to the present disclosure.

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technologies (part or all of the products or methods) employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technique employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technique employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technique shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0013] (Configuration of Information Processing System 1) The configuration of the information processing system 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing system 1. As shown in FIG. 1, the information processing system 1 includes a division unit 11 and an extraction unit 12. The division unit 11 is an example of a configuration that realizes division means. The extraction unit 12 is an example of a configuration that realizes extraction means. As an example, the information processing system 1 may be configured by multiple computers or by a single computer. The division unit 11 and the extraction unit 12 may each be realized by at least one processor executing a program.

[0014] The dividing unit 11 divides a three-dimensional region containing point cloud data in a virtual space into multiple partial regions. Here, the point cloud data is a set of points associated with three-dimensional coordinates. For example, the point cloud data may be a set of points corresponding to an object existing in real space. Furthermore, the point cloud data may be a set of points corresponding to the surface of such an object. As an example, point cloud data corresponding to an object existing in real space can be generated using a distance sensor that measures the distance to the object. Furthermore, as an example, the point cloud data does not necessarily have to correspond to an object existing in real space, but may also be a set of points representing a virtual object. Furthermore, the point cloud data may be generated in real time, or may be generated in advance and stored in memory.

[0015] A three-dimensional region including point cloud data is defined as a three-dimensional region that includes at least each point that constitutes the point cloud data. The shape of the three-dimensional region may be, but is not limited to, a rectangular parallelepiped, a cube, a sphere, a hemisphere, an ellipsoid, a semi-ellipsoid, or the like. Partial regions obtained by dividing a three-dimensional region are also three-dimensional regions. The multiple partial regions obtained by division may have the same shape, or at least two partial regions may have different shapes.

[0016] The extraction unit 12 extracts second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial region of the point cloud data. The shape indicated by the first point cloud data is the shape of a virtual object represented by the first point cloud data. For example, the shape indicated by the first point cloud data may be the shape of a mesh constructed from the first point cloud data. An example of the shape evaluation result is a result of evaluating the complexity of the shape. Here, for example, the shape indicated by the first point cloud data may be deemed more complex the greater the difference between the shape and the planar shape. Another example of the shape evaluation result is a result of evaluating shape categories. However, the shape evaluation result is not limited to these.

[0017] By extracting the second point cloud data from the first point cloud data for a certain partial region, the amount of the second point cloud data is equal to or less than the amount of the first point cloud data. Furthermore, it is desirable that the amount of the second point cloud data is less than the amount of the first point cloud data for at least one partial region among the plurality of partial regions. In other words, the extraction unit 12 extracts the second point cloud data from the first point cloud data for each partial region, thereby reducing the amount of data from the "original point cloud data" consisting of the plurality of first point cloud data. Hereinafter, the point cloud data consisting of the plurality of second point cloud data will also be referred to as "point cloud data after data amount reduction."

[0018] (Effects of Information Processing System 1) As described above, the information processing system 1 is configured to include the dividing unit 11 and the extracting unit 12. In this manner, the second point cloud data for each partial region is extracted based on the evaluation result of the shape indicated by the first point cloud data. Therefore, it is expected that the degree to which the quality of the shape indicated by the second point cloud data is reduced from the quality of the shape indicated by the first point cloud data is reduced based on the shape evaluation result. As a result, the information processing system 1 can achieve the effect of reducing the quality of a virtual space reproduced based on point cloud data after data volume reduction. Note that the "virtual space reproduced based on point cloud data" refers to "a virtual space reproduced from point cloud data or by processing point cloud data." An example of the "virtual space reproduced by processing point cloud data" includes, but is not limited to, "a virtual space reproduced from a mesh constructed from point cloud data."

[0019] (Configuration of information processing device 10) The configuration of the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the information processing device 10. As shown in Fig. 2, the information processing device 10 includes a division unit 11 and an extraction unit 12. The division unit 11 and the extraction unit 12 have been described above, and therefore detailed description thereof will not be repeated. As an example, the information processing device 10 may be configured by a single computer or by multiple computers.

[0020] (Effects of Information Processing Device 10) As described above, the information processing device 10 employs a configuration including the above-described division unit 11 and extraction unit 12. Therefore, the information processing device 10 can obtain the same effects as the information processing system 1.

[0021] (Flow of information processing method S1) The flow of information processing method S1 will be described with reference to Fig. 3. Fig. 3 is a flow diagram showing the flow of information processing method S1. As shown in Fig. 3, information processing method S1 includes a division process S11 and an extraction process S12.

[0022] In the division process S11, at least one processor (for example, the division unit 11) divides a three-dimensional area including point cloud data in a virtual space into a plurality of partial areas.

[0023] In extraction process S12, at least one processor (for example, extraction unit 12) extracts second point cloud data from the first point cloud data based on the evaluation result of the shape indicated by the first point cloud data included in each partial area of ​​the point cloud data.

[0024] (Effects of Information Processing Method S1) As described above, the information processing method S1 includes the division process S11 and the extraction process S12. Therefore, the information processing method S1 can achieve the same effects as the information processing system 1.

[0025] Second Exemplary Embodiment A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0026] (Configuration of Information Processing System 1A) The configuration of the information processing system 1A will be described with reference to FIG. 4. FIG. 4 is a block diagram showing the configuration of the information processing system 1A. As shown in FIG. 4, the information processing system 1A includes a point cloud data acquisition unit 13, a complexity calculation unit 14, an extraction rate determination unit 15, and a point cloud data output unit 16 in addition to the division unit 11 and extraction unit 12 included in the information processing system 1. The complexity calculation unit 14 and the extraction rate determination unit 15 are examples of configurations that realize at least a portion of the extraction means. The information processing system 1A may be configured by multiple computers or a single computer. Furthermore, the division unit 11, the extraction unit 12, the point cloud data acquisition unit 13, the complexity calculation unit 14, the extraction rate determination unit 15, and the point cloud data output unit 16 may each be realized by at least one processor executing a program.

[0027] The point cloud data acquisition unit 13 acquires point cloud data. Details of the point cloud data are as described in the exemplary embodiment 1, and therefore detailed description will not be repeated. For example, the point cloud data acquisition unit 13 may acquire point cloud data stored in its own device or another device, or may acquire point cloud data generated in real time.

[0028] The dividing unit 11 is configured in the same manner as in the first exemplary embodiment.

[0029] The extraction unit 12 is configured similarly to the first exemplary embodiment, but is also configured as follows. The extraction unit 12 refers to the complexity of the shape indicated by the first point cloud data included in each partial region as an evaluation result of the shape indicated by the first point cloud data. For example, the extraction unit 12 may extract the second point cloud data such that the amount of second point cloud data increases as the shape of the first point cloud data increases in complexity. This allows the second point cloud data to be extracted with high accuracy according to the complexity of the shape of each partial region in the original point cloud data. Furthermore, increasing the amount of second point cloud data as the shape of the first point cloud data increases in complexity provides the following effect. That is, by reducing the amount of data reduction for parts with complex shapes where data reduction has a large impact on quality and increasing the amount of data reduction for parts with simple shapes where data reduction has a small impact on quality, the amount of data reduction can be efficiently reduced while suppressing overall quality degradation.

[0030] For example, the complexity referred to by the extraction unit 12 is calculated by the complexity calculation unit 14. Examples of a method for the complexity calculation unit 14 to calculate the complexity include, but are not limited to, the following two calculation methods.

[0031] As a first calculation method, the complexity calculation unit 14 may evaluate the complexity of the shape indicated by the first point cloud data based on the normal at each of the multiple points included in the first point cloud data relative to the surface indicated by the first point cloud data. This allows for accurate evaluation of the complexity of the shape indicated by the first point cloud data. More specifically, the complexity calculation unit 14 may calculate, as the complexity, the variance of the distribution of the dot product between the normal at an arbitrary point included in the first point cloud data and the normal at each of the multiple other points. As an example, the distribution of the dot product of the normal for an arbitrary point i included in the first point cloud data can be calculated using the following equation (1). Note that in this specification, subscripts such as "i" in each equation below will also be written as "_i" using an underscore "_". Furthermore, superscripts such as "r" in each equation will also be written as "^r" using a hat "^".

[0032] In formula (1), n_i is a vector indicating the normal at point i with respect to the surface indicated by the first point cloud data. Furthermore, point j^r indicates a point in the first point cloud data that is within a distance r from point i. n_j^r is a vector indicating the normal at point j^r. n_(i,j)(r) indicates the dot product of normal n_i and normal n_j^r, and is normalized so that its magnitude is between 0 and 1. m_i indicates the average value of the dot products n_(i,j)(r) calculated between point i and k points j^r. The distribution of standardized dot products can be obtained by dividing the difference of the dot product n_(i,j)(r) from the average value m_i by the standard deviation.

[0033] FIG. 5 is a diagram showing an example of the distribution of dot products of normals in a certain partial region. In FIG. 5, distribution diagram G1 shows an example of the distribution of dot products of normals when the first point cloud data indicates a substantially planar shape. The normals at each point on the plane face in the same direction. Therefore, the dot products of normals between any point i included in the first point cloud data in the partial region and each of k other points j^r are mostly nearly the same, being the average value. Note that distribution diagram G1 also shows a few dot products that deviate from the average value. This indicates that the shape indicated by the first point cloud data is close to a plane but not strictly a plane.

[0034] Furthermore, distribution diagram G2 shows an example of the distribution of the dot product of normals when the surface represented by the first point cloud data is approximately spherical. The normals at each point on the spherical surface point in different directions. Furthermore, the greater the distance between two points on the spherical surface, the larger the angle between the normals. Therefore, the dot product of the normals between any point i and k other points j^r is distributed in the range from -1 to 1.

[0035] As shown in the distribution maps G1 and G2, the smaller the variance of the dot product of the normals, the closer the object is to a plane. Therefore, the variance of the dot product of the normals can be used as a measure of complexity, with the larger the variance, the more complex the object. The heat map G3 calculates the complexity of each subregion of a three-dimensional cubic area based on the variance of the distribution of the dot products of the normals, and displays the magnitude of the calculated complexity in a heat map. In the heat map G3, the darker the grayscale color, the greater the complexity. For example, the complexity of the subregions located between the edges and faces of a cubic shape is higher than that of the subregions that include the edges or faces. This is because the direction of the normals changes significantly between the edges and faces.

[0036] As a second calculation method, the complexity calculation unit 14 may evaluate the complexity of the shape indicated by the first point cloud data based on the results of principal component analysis of the first point cloud data. This allows the complexity of the shape indicated by the first point cloud data to be evaluated with high accuracy. For example, the complexity based on the results of principal component analysis can be calculated using the following equation (2).

[0037] In equation (2), λ_1 represents the largest eigenvalue, λ_2 represents the second largest eigenvalue, and λ_3 represents the third largest eigenvalue. In a three-dimensional subregion, the smaller λ_3 corresponding to the third dimension is relative to λ_1 and λ_2 corresponding to the two dimensions, the closer the region is to a plane. Therefore, the index c_n obtained by dividing λ_3 by the sum of λ_1 to λ_3 can be used as a complexity index, where the larger the index, the more complex the region.

[0038] Furthermore, the extraction unit 12 extracts second point cloud data from the first point cloud data at an extraction rate determined by the extraction rate determination unit 15. The extraction rate determination unit 15 determines the extraction rate of the second point cloud data based on the evaluation result so that the shape indicated by the second point cloud data satisfies a predetermined quality. This ensures the quality of each partial region of the data point cloud after data volume reduction.

[0039] Here, the extraction rate refers to the ratio of the amount of second point cloud data to the amount of first point cloud data. Reducing the extraction rate corresponds to increasing the amount of data reduction. Furthermore, an example of the quality of the shape represented by the second point cloud data is the quality of the mesh formed by the second point cloud data. For example, the quality of the shape represented by the second point cloud data can be evaluated based on the degree of degradation from the shape represented by the first point cloud data from which the second point cloud data was extracted. An example of the degree of degradation is the chamfer distance (CD) value. More specifically, the degree of degradation can be calculated as the CD (chamfer distance) value of point cloud data obtained by separately sampling from the mesh formed by the second point cloud data relative to point cloud data obtained by separately sampling from the mesh formed by the first point cloud data. In this case, the smaller the CD value, the less degradation there is, i.e., the higher the quality. However, the quality of the shape represented by the second point cloud data is not limited to the CD value.

[0040] For example, the extraction rate determination unit 15 may determine the extraction rate based on relationship information that defines the relationship between the quality of the shape indicated by the second point cloud data and the extraction rate for each level of complexity. FIG. 6 is a diagram illustrating an example of relationship information indicating the relationship between mesh quality and extraction rate. In FIG. 6, the vertical axis represents mesh quality, and the horizontal axis represents extraction rate. In this example, four levels of complexity are set, designated Levels 1, 2, 3, and 4 in order of increasing complexity. Note that when the complexity is calculated as a numerical value, the complexity may be classified into one of four levels depending on the range. Relationship information L1 to L4 indicate the relationship between mesh quality and extraction rate corresponding to Levels 1 to 4 of complexity, respectively. Each of the relationship information L1 to L4 is a downward-sloping line, indicating that the quality decreases as the extraction rate decreases. Furthermore, the rate of change increases in the order of the relationship information L1, L2, L3, and L4. In other words, the higher the complexity, the greater the degree of quality degradation with a decrease in extraction rate. Such relationship information can be generated in advance. For example, the relationship information may be generated by determining the quality of the second point cloud data extracted while changing the extraction rate for multiple first point cloud data used for learning, and then determining an approximate curve for each level of complexity of the first point cloud data.

[0041] The following three determination methods can be given as examples of the method by which the extraction rate determination unit 15 determines the extraction rate using such relationship information.

[0042] In the first determination method, the extraction rate determination unit 15 may determine the extraction rate by applying a common predetermined quality to each partial region. This ensures that the point cloud data after data volume reduction satisfies the common predetermined quality for all partial regions. FIG. 7 is a diagram schematically illustrating a method for determining the extraction rate using the common predetermined quality. As shown in FIG. 7 , when the common predetermined quality V0 is given, the extraction rate determination unit 15 determines the extraction rate d1 corresponding to the predetermined quality V0 using the relationship information L1 for partial regions where the shape complexity level indicated by the first point cloud data is level 1. Similarly, the extraction rate determination unit 15 determines the extraction rate d2, d3, or d4 corresponding to the predetermined quality V0 using the relationship information L2, L3, or 4 for partial regions where the shape complexity level indicated by the first point cloud data is level 2, 3, or 4.

[0043] In the second determination method, the extraction rate determination unit 15 may determine the predetermined quality for each partial region based on the evaluation results of the shape indicated by the first point cloud data. For example, the extraction rate determination unit 15 may determine a higher predetermined quality for each partial region based on the complexity of the shape indicated by the first point cloud data. In this case, the extraction rate determination unit 15 may determine, for each partial region, an extraction rate corresponding to the predetermined quality for the level of complexity using relationship information corresponding to the level of complexity for that partial region, as shown in FIG. 6 . Here, it can be assumed that for partial regions with simple shapes where data reduction has little impact on quality, even if quality is reduced after data reduction, degradation is not noticeable. This allows for more efficient data reduction in partial regions with simpler shapes, while reducing the amount of data reduction to suppress quality degradation in partial regions with complex shapes where the impact on quality is significant.

[0044] In the third determination method, the extraction rate determination unit 15 may determine the predetermined quality and extraction rate for each partial region so that the quality of the overall shape represented by the plurality of second point cloud data is higher and the total amount of the plurality of second point cloud data does not exceed a threshold. This allows the total amount of point cloud data after data volume reduction to be kept below an allowable threshold while suppressing degradation in the quality of the overall shape represented by the point cloud data after data volume reduction. For example, in applications where the point cloud data after data volume reduction is transferred to another device, the threshold may be determined based on the allowable data transfer amount. In this case, the predetermined quality and extraction rate can be determined, for example, by solving the optimization problem shown in the following equation (3).

[0045] The first line of Equation (3) indicates that the sum of the degradation levels of the second point cloud data extracted from the first point cloud data according to the extraction rate for each partial region should be minimized. Here, "reducing the sum of the degradation levels for each partial region" is an example of "enhancing the quality of the overall shape represented by the plurality of second point cloud data." The CD value described above may be used as the degradation level, but is not limited to this. Furthermore, the second line of Equation (3) indicates that the total amount of the second point cloud data extracted from the first point cloud data according to the extraction rate for each partial region should be equal to or less than a threshold value.

[0046] 8 is a diagram schematically illustrating an example of a determination method for increasing the overall shape quality of a plurality of second point cloud data while keeping the total amount below a threshold. As shown in FIG. 8 , for a partial region with a complexity level of level 1, a predetermined quality V1 and an extraction rate d1 that satisfy the relationship information L1 are determined. For a partial region with a complexity level of level 2, a predetermined quality V2 and an extraction rate d2 that satisfy the relationship information L2 are determined. For a partial region with a complexity level of level 3, a predetermined quality V3 and an extraction rate d4 that satisfy the relationship information L3 are determined. This is because, by solving equation (3), the predetermined qualities V1 to V4 and the extraction rates d1 to d4 are determined so that the sum of the predetermined qualities V1 to V4 is larger and the total amount of second point cloud data calculated based on the extraction rates d1 to d4 is below a threshold.

[0047] (Flow of Information Processing Method S1A) The information processing system 1A configured as described above executes the information processing method S1A. Fig. 9 is a flow diagram showing the flow of the information processing method S1A. As shown in Fig. 9, the information processing method S1A includes steps S101 to S106.

[0048] Step S101 is an example of a point cloud data acquisition process. In step S101, the point cloud data acquisition unit 13 acquires point cloud data.

[0049] Step S102 is an example of a division process. In step S102, the division unit 11 divides a three-dimensional area including point cloud data in the virtual space into a plurality of partial areas.

[0050] Step S103 is an example of evaluation processing. In step S103, the complexity calculation unit 14 calculates the complexity as an evaluation result of the shape indicated by the first point cloud data in each partial region. As described above, the complexity calculation unit 14 may calculate the complexity based on the distribution of the dot product of normals in each partial region, or may calculate the complexity based on the results of principal component analysis.

[0051] Step S104 is an example of an extraction rate determination process. In step S104, the extraction rate determination unit 15 determines, for each partial region, an extraction rate of the second point cloud data such that the shape indicated by the second point cloud data satisfies a predetermined quality, depending on the complexity of the partial region. As described above, the extraction rate determination unit 15 may determine the extraction rate using relationship information that defines the relationship between mesh quality and extraction rate for each level of complexity. Also, as described above, the extraction rate determination unit 15 may use a common predetermined quality to determine, for each partial region, an extraction rate corresponding to the common predetermined quality that satisfies the relationship information according to the complexity of the partial region. Also, as described above, the extraction rate determination unit 15 may determine, for each partial region, an extraction rate corresponding to a predetermined quality according to the complexity that satisfies the relationship information according to the complexity of the partial region. Also, as described above, the extraction rate determination unit 15 may determine the predetermined quality and the extraction rate so as to increase the quality of the overall shape indicated by the plurality of second point cloud data and so that the total amount of the plurality of second point cloud data is equal to or less than a threshold.

[0052] Step S105 is an example of extraction processing. In step S105, the extraction unit 12 extracts second point cloud data from the first point cloud data in each partial region according to the extraction rate determined in step S104.

[0053] Step S106 is an example of a point cloud data output process. In step S106, the point cloud data output unit 16 outputs the point cloud data after data volume reduction, which is composed of the extracted plurality of second point cloud data. Note that, here, "output" may mean storing the data in a storage device, or displaying the shape indicated by the point cloud data after data volume reduction on a display device. Furthermore, "output" may mean transmitting the data via a network.

[0054] FIG. 10 is a diagram illustrating point cloud data after data volume reduction according to this exemplary embodiment in comparison with a comparative example.

[0055] Images G21 to G24 shown in FIG. 10 all show point cloud data generated by reducing the amount of data from point cloud data A and B obtained from "http: / / redwood-data.org / ," and are rendered based on meshes formed from the point cloud data after data reduction. Image G21 shows point cloud data after data reduction using information processing method S1A from point cloud data A corresponding to an object such as a chair arranged in a three-dimensional area. Image G22 shows point cloud data after data reduction using FPS (Farthest Point Sampling) from the same point cloud data A as image G21. It can be seen that image G21 has higher quality, for example, in the legs of the chair, compared to image G22.

[0056] Image G23 shows point cloud data after the amount of data has been reduced by information processing method S1A from point cloud data B corresponding to an object such as a table placed in a three-dimensional area. Image G24 shows point cloud data after the amount of data has been reduced by FPS from the same point cloud data B as image G23. It can be seen that image G23 has higher quality of the object placed on, for example, a table, compared to image G24.

[0057] FIG. 11 shows an example of measurement results of extraction rates and CD values ​​in data volume reduction. In FIG. 11, graph GA shows the measurement results for point cloud data A, similar to FIG. 10. The extraction rate here refers to the ratio of the total amount of point cloud data after data volume reduction to the total amount of point cloud data A. Measurement result R21 shows the measurement results when information processing method S1A is used, and measurement result R22 shows the measurement results when FPS is used. Furthermore, graph GB in FIG. 11 shows the measurement results for point cloud data B, similar to FIG. 10. The extraction rate here refers to the ratio of the total amount of point cloud data after data volume reduction to the total amount of point cloud data B. Measurement result R23 shows the measurement results when information processing method S1A is used, and measurement result R24 shows the measurement results when FPS is used.

[0058] 11, in the measurement results R22 and R24 when FPS is used, the CD value increases as the extraction rate decreases. In other words, the quality decreases as the data reduction rate increases. In contrast, in the measurement results R21 and R23 according to this exemplary embodiment, the increase in the CD value is suppressed even when the extraction rate decreases. In other words, it can be seen that the quality degradation is suppressed even when the data reduction rate increases.

[0059] As described above, the information processing system 1A employs a configuration in which the extraction unit 12 refers to the degree of complexity of the shape indicated by the first point cloud data as an evaluation result of the shape indicated by the first point cloud data. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A can also provide an effect of being able to accurately extract second point cloud data in accordance with the complexity of the shape of each partial region in the original point cloud data.

[0060] Furthermore, in the information processing system 1A, the extraction rate determination unit 15 determines the extraction rate of the second point cloud data such that the shape indicated by the second point cloud data satisfies a predetermined quality in accordance with the evaluation result of the shape indicated by the first point cloud data. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A has the effect of guaranteeing the quality of each partial region of the data point cloud after data volume reduction.

[0061] Furthermore, the information processing system 1A employs a configuration in which the extraction rate determination unit 15 determines the predetermined quality based on the evaluation results of the shape indicated by the first point cloud data for each partial region. Here, the impact of the degree of data reduction on quality may vary depending on the evaluation results of the shape indicated by the first point cloud data. For example, even if the data volume is reduced to the same extent, the impact on quality may be smaller for simpler shapes and larger for more complex shapes. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A also provides the effect of ensuring appropriate quality for each partial region of the point cloud data after data volume reduction, based on the evaluation results of the shape indicated by the original first point cloud data.

[0062] Furthermore, in the information processing system 1A, the extraction rate determination unit 15 is configured to determine a predetermined quality and an extraction rate for each partial region so that the quality of the overall shape represented by the plurality of second point cloud data is higher and so that the total amount of the plurality of second point cloud data does not exceed a threshold. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A has the effect of suppressing a deterioration in the quality of the overall shape represented by the point cloud data after data volume reduction while keeping the total amount of point cloud data below an allowable threshold.

[0063] Furthermore, in the information processing system 1A, the complexity calculation unit 14 evaluates the complexity of the shape indicated by the first point cloud data based on the normal of each of the points included in the first point cloud data with respect to the surface indicated by the first point cloud data. Therefore, in addition to the effects achieved by the information processing system 1, the information processing system 1A can accurately evaluate the complexity of the shape indicated by the first point cloud data.

[0064] Furthermore, the information processing system 1A is configured such that the complexity calculation unit 14 evaluates the complexity of the shape indicated by the first point cloud data based on the results of principal component analysis of the first point cloud data. Therefore, in addition to the effects of the information processing system 1, the information processing system 1A can accurately evaluate the complexity of the shape indicated by the first point cloud data.

[0065] (Modification 1) In each exemplary embodiment, an example has been described in which the shape indicated by the first point cloud data or the second point cloud data is the shape of a mesh configured by the point cloud data, but this is not limiting.

[0066] 4 as exemplary embodiment 2 may be arranged in one computer, or may be arranged in multiple computers. A modification in which each functional block is arranged in multiple computers will be described with reference to FIG. 12.

[0067] FIG. 12 is a diagram showing a functional block configuration of an information processing system 1B in which each functional block is arranged in a server 2, a transmitting device 3, and a receiving device 4. As shown in FIG. 12, the information processing system 1B includes a server 2, a transmitting device 3, and a receiving device 4. The transmitting device 3 includes a point cloud data acquisition unit 13 and a communication unit 301. The point cloud data acquisition unit 13 acquires point cloud data, and the communication unit 301 transmits the point cloud data to the server 2. The server 2 includes a communication unit 201, a dividing unit 11, a complexity calculation unit 14, an extraction rate determination unit 15, and an extraction unit 12. The dividing unit 11 divides a three-dimensional region including the point cloud data received by the communication unit 201 from the transmitting device 3 into multiple partial regions. The complexity calculation unit 14, the extraction rate determination unit 15, and the extraction unit 12 are configured as described above, thereby extracting second point cloud data for each partial region. The communication unit 201 transmits a plurality of second point cloud data, i.e., point cloud data after data volume reduction, to the receiving device 4. The receiving device 4 includes a communication unit 401 and a point cloud data output unit 16. The communication unit 401 receives the point cloud data after data volume reduction from the server 2, and the point cloud data output unit 16 outputs the point cloud data. This configuration is effective when there is ample capacity for transfer volume between the transmitting device 3 and the server 2, but there is a limit to the transfer volume between the server 2 and the receiving device 4.

[0068] (Modification 3) Another modification in which the functional blocks shown in FIG. 4 as exemplary embodiment 2 are arranged in a plurality of computers will be described with reference to FIG.

[0069] FIG. 13 is a diagram showing the functional block configuration of an information processing system 1C in which each functional block is arranged in a transmitting device 3C and a receiving device 4. As shown in FIG. 13, the information processing system 1C includes a transmitting device 3C and a receiving device 4. The transmitting device 3C includes a point cloud data acquisition unit 13, a division unit 11, a complexity calculation unit 14, an extraction rate determination unit 15, an extraction unit 12, and a communication unit 301. The point cloud data acquisition unit 13, the complexity calculation unit 14, the extraction rate determination unit 15, and the extraction unit 12 are configured as described above to extract second point cloud data for each partial region. The communication unit 301 transmits multiple pieces of second point cloud data, i.e., point cloud data after data volume reduction, to the receiving device 4. The receiving device 4 is configured as described above. That is, the communication unit 401 receives the point cloud data after data volume reduction from the transmitting device 3, and the point cloud data output unit 16 outputs the point cloud data. This configuration is effective when there is a limit to the amount of data transferred between the transmitting device 3 and the receiving device 4.

[0070] (Application Example) For example, the information processing systems 1B and 1C can be applied as a remote monitoring system. As an example, the transmitting devices 3 and 3C acquire point cloud data of a monitored space in real time and transmit the point cloud data after data volume reduction to the receiving device 4 via the server 2 or directly. The receiving device 4 displays the monitoring video represented by the point cloud data after data volume reduction on a display device. The display device may be, for example, a virtual reality device such as a head-mounted display. This makes it possible to reduce the data transfer volume of the point cloud data representing the monitored space while suppressing degradation in the quality of the monitoring video reproduced in the monitored area.

[0071] Furthermore, for example, the information processing systems 1B and 1C can be applied as systems for distributing replay footage of interesting scenes in live sports broadcasts. As an example, the transmitting devices 3 and 3C acquire point cloud data of interesting scenes in live sports broadcasts and transmit the point cloud data after data volume reduction to the receiving device 4 via the server 2 or directly. The receiving device 4 distributes the replay footage based on the point cloud data after data volume reduction to a distribution terminal (not shown). This makes it possible to reduce the data transfer volume of the replay footage of interesting scenes in live sports broadcasts while suppressing degradation in the quality of the replay footage played on the distribution terminal.

[0072] [Example of implementation by software] Some or all of the functions of each device constituting the information processing systems 1, 1A, 1B, and 1C, and the information processing device 10 (hereinafter also referred to as "each of the above-mentioned devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0073] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 14. Figure 14 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0074] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0075] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0076] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0077] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0078] Furthermore, the functions of each of the devices may be realized by a single processor provided in a single computer, by multiple processors provided in a single computer working in cooperation, or by multiple processors provided in each of multiple computers working in cooperation. Furthermore, the programs for causing each of the devices to realize the functions may be stored in a single memory provided in a single computer, or may be distributed and stored in multiple memories provided in a single computer, or may be distributed and stored in multiple memories provided in each of multiple computers.

[0079] [Appendix 1] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0080] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0081] (Appendix A1) An information processing system comprising: a division means for dividing a three-dimensional area including point cloud data in a virtual space into a plurality of partial areas; and an extraction means for extracting second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0082] (Appendix A2) The information processing system according to appendix A1, wherein the extraction means refers to a complexity degree obtained by evaluating the complexity of the shape as the evaluation result.

[0083] (Supplementary Note A3) The information processing system according to Supplementary Note A1 or A2, wherein the extraction unit determines an extraction rate of the second point cloud data such that the shape indicated by the second point cloud data satisfies a predetermined quality, according to the evaluation result.

[0084] (Supplementary Note A4) The information processing system according to Supplementary Note A3, wherein the extraction means determines the predetermined quality for each partial region according to the evaluation result.

[0085] (Appendix A5) The information processing system according to Appendix A3, wherein the extraction means determines the predetermined quality and the extraction rate for each partial region so that the quality of the overall shape indicated by the plurality of second point cloud data is higher and so that the total amount of the plurality of second point cloud data does not exceed a threshold.

[0086] (Supplementary Note A6) The information processing system according to Supplementary Note A2, wherein the extraction means evaluates the complexity based on a normal at each of a plurality of points included in the first point cloud data with respect to a surface indicated by the first point cloud data.

[0087] (Supplementary Note A7) The information processing system according to Supplementary Note A2, wherein the extraction means evaluates the complexity based on a result of a principal component analysis of the first point cloud data.

[0088] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0089] (Appendix B1) An information processing method including: a division process in which at least one processor divides a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas; and an extraction process in which the at least one processor extracts second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0090] (Supplementary Note B2) The information processing method according to Supplementary Note B1, wherein in the extraction process, the at least one processor refers to a complexity degree obtained by evaluating the complexity of the shape as the evaluation result.

[0091] (Supplementary Note B3) The information processing method according to Supplementary Note B1 or B2, wherein in the extraction process, the at least one processor determines an extraction rate of the second point cloud data such that the shape indicated by the second point cloud data satisfies a predetermined quality, according to the evaluation result.

[0092] (Supplementary Note B4) The information processing method according to Supplementary Note B3, wherein in the extraction process, the at least one processor determines the predetermined quality for each partial region according to the evaluation result.

[0093] (Appendix B5) The information processing method described in Appendix B3, wherein in the extraction process, the at least one processor determines the predetermined quality and the extraction rate for each partial region so that the quality of the overall shape represented by the plurality of second point cloud data is higher and so that the total amount of the plurality of second point cloud data does not exceed a threshold.

[0094] (Supplementary Note B6) The information processing method according to Supplementary Note B2, wherein in the extraction process, the at least one processor evaluates the complexity based on a normal at each of a plurality of points included in the first point cloud data with respect to a surface indicated by the first point cloud data.

[0095] (Supplementary Note B7) The information processing method according to Supplementary Note B2, wherein in the extraction process, the at least one processor evaluates the complexity based on a result of a principal component analysis of the first point cloud data.

[0096] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0097] (Appendix C1) An information processing program that causes a computer to function as an information processing system, the information processing program causing the computer to function as: a division means that divides a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas; and an extraction means that extracts second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0098] (Supplementary Note C2) The information processing program according to Supplementary Note C1, wherein the extraction means refers to a complexity degree obtained by evaluating the complexity of the shape as the evaluation result.

[0099] (Supplementary Note C3) The information processing program according to Supplementary Note C1 or C2, wherein the extraction means determines an extraction rate of the second point cloud data such that a shape indicated by the second point cloud data satisfies a predetermined quality, according to the evaluation result.

[0100] (Supplementary Note C4) The information processing program according to Supplementary Note C3, wherein the extraction means determines the predetermined quality for each partial region according to the evaluation result.

[0101] (Appendix C5) The information processing program according to Appendix C3, wherein the extraction means determines the predetermined quality and the extraction rate for each partial region so that the quality of the overall shape indicated by the plurality of second point cloud data is higher and so that the total amount of the plurality of second point cloud data does not exceed a threshold.

[0102] (Supplementary Note C6) The information processing program according to Supplementary Note C2, wherein the extraction means evaluates the complexity based on a normal at each of a plurality of points included in the first point cloud data with respect to a surface indicated by the first point cloud data.

[0103] (Supplementary Note C7) The information processing program according to Supplementary Note C2, wherein the extraction means evaluates the complexity based on a result of a principal component analysis of the first point cloud data.

[0104] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0105] (Appendix D1) An information processing system comprising at least one processor, the at least one processor executing a division process of dividing a three-dimensional area including point cloud data in a virtual space into a plurality of partial areas, and an extraction process of extracting second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0106] The information processing system may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0107] (Supplementary Note D2) The information processing system according to Supplementary Note D1, wherein in the extraction process, the at least one processor refers to a complexity degree that evaluates the complexity of the shape as the evaluation result.

[0108] (Appendix D3) The information processing system according to appendix D1 or D2, wherein in the extraction process, the at least one processor determines an extraction rate of the second point cloud data such that the shape indicated by the second point cloud data satisfies a predetermined quality, according to the evaluation result.

[0109] (Supplementary Note D4) The information processing system according to Supplementary Note D3, wherein in the extraction process, the at least one processor determines the predetermined quality for each partial region according to the evaluation result.

[0110] (Appendix D5) The information processing system described in Appendix D3, wherein in the extraction process, the at least one processor determines the predetermined quality and the extraction rate for each partial region so that the quality of the overall shape represented by the plurality of second point cloud data is higher and so that the total amount of the plurality of second point cloud data does not exceed a threshold.

[0111] (Supplementary Note D6) The information processing system according to Supplementary Note D2, wherein in the extraction process, the at least one processor evaluates the complexity based on a normal at each of a plurality of points included in the first point cloud data with respect to a surface indicated by the first point cloud data.

[0112] (Supplementary Note D7) The information processing system according to Supplementary Note D2, wherein in the extraction process, the at least one processor evaluates the complexity based on a result of a principal component analysis of the first point cloud data.

[0113] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0114] (Appendix E1) A non-transitory recording medium having recorded thereon an information processing program that causes a computer to function as an information processing system, the information processing program causing the computer to execute: a division process that divides a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas; and an extraction process that extracts second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

[0115] REFERENCE SIGNS LIST 1, 1A, 1B, 1C Information processing system 2 Server 3, 3C Transmission device 4 Reception device 10 Information processing device 11 Division unit 12 Extraction unit 13 Point cloud data acquisition unit 14 Complexity calculation unit 15 Extraction rate determination unit 16 Point cloud data output unit 201, 301, 401 Communication unit

Claims

1. An information processing system comprising: a division means for dividing a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas; and an extraction means for extracting second point cloud data from the first point cloud data in accordance with an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

2. The information processing system according to claim 1, wherein said extraction means refers to a degree of complexity that evaluates the complexity of said shape as said evaluation result.

3. The information processing system according to claim 1 or 2, wherein the extraction means determines an extraction rate of the second point cloud data such that the shape indicated by the second point cloud data satisfies a predetermined quality, depending on the evaluation result.

4. The information processing system according to claim 3, wherein the extraction means determines the predetermined quality for each partial region in accordance with the evaluation result.

5. The information processing system according to claim 3, wherein the extraction means determines the predetermined quality and the extraction rate for each partial region so that the quality of the overall shape represented by the plurality of second point cloud data is higher and so that the total amount of the plurality of second point cloud data does not exceed a threshold value.

6. The information processing system according to claim 2, wherein the extraction means evaluates the complexity based on a normal at each of a plurality of points included in the first point cloud data with respect to a surface indicated by the first point cloud data.

7. The information processing system according to claim 2, wherein the extraction means evaluates the complexity based on a result of a principal component analysis of the first point cloud data.

8. An information processing device comprising: a division means for dividing a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas; and an extraction means for extracting second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

9. An information processing method comprising: a division process in which at least one processor divides a three-dimensional area containing point cloud data in a virtual space into a plurality of partial areas; and an extraction process in which the at least one processor extracts second point cloud data from the first point cloud data according to an evaluation result of a shape indicated by first point cloud data included in each partial area of ​​the point cloud data.

Images