Information processing apparatus, information processing method, and program

Abstract

To reduce the occurrence of a situation where a CG model that essentially enters a field of vision is deleted, while reducing human's manual labor.SOLUTION: An information processing apparatus acquires the geometric inclusion relationship between CG models constituting scenes represented by CG, and acquires information on operations performed by a user. The information processing apparatus determines, on the basis of the inclusion relationship and operation information, a three-dimensional model to be processed to be simplified, from among three-dimensional models constituting the scenes, and simplification processing to be executed on the target three-dimensional model, and executes the determined simplification processing on the three-dimensional model to be processed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to information processing technology used for virtual reality, mixed reality, augmented reality, and the like. [Background technology]

[0002] In recent years, systems have been developed that provide users with XR, such as virtual reality (VR), mixed reality (MR), and augmented reality (AR). XR is a general term for technologies such as VR, MR, and AR that combine the real world and the virtual world to create new experiences.

[0003] In fields that handle 3D CG models such as XR, culling of CG models is known as a technique for improving the processing efficiency of CG rendering. For example, culling CG models that are outside the field of view or hidden by an obstructing object can be used to improve the processing efficiency of CG rendering, making it possible to achieve nearly real-time CG rendering. Furthermore, Patent Document 1 discloses a technique in which a viewpoint position is determined in advance, and CG models that are not within the field of view from that viewpoint are deleted in advance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-72916 A Summary of the Invention [Problem to be solved by the invention]

[0005] When culling CG models as described above, it is necessary to determine which CG models to cull for each frame, which generates overhead, so that data that can be deleted must be deleted in advance. However, which CG models should be deleted and how they should be deleted differ depending on the use scene, so human judgment is required, resulting in a lot of manual work. On the other hand, in the case of the technology described in Patent Document 1, CG models that are not in the field of view are deleted in advance based on a predetermined viewpoint position, so manual work by humans can be reduced to some extent. However, in XR such as VR, MR, and AR, the degree of freedom and range of viewpoints that can be taken at runtime is large, so even CG models that could be in the field of view may be deleted in advance.

[0006] Therefore, an object of the present invention is to reduce the amount of manual work performed by humans while reducing the occurrence of a situation in which a CG model that may be within the field of view is deleted. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a relationship acquisition means for acquiring a geometric inclusion relationship between three-dimensional models constituting a scene represented in computer graphics, an operation acquisition means for acquiring operation information by a user, a determination means for determining a three-dimensional model to be subjected to simplification processing among the three-dimensional models constituting the scene based on the inclusion relationship and the operation information, and a processing means for applying the simplification processing determined by the determination means to the three-dimensional model to be processed. Effect of the Invention

[0008] According to the present invention, it is possible to reduce manual work, and to reduce the occurrence of a situation in which a CG model that may be within the field of view is deleted. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to a first embodiment. [Diagram 2] 4 is a flowchart of information processing according to the first embodiment. [Diagram 3] 13 is a flowchart of an inclusion relationship acquisition process. [Figure 4] 13A to 13C are diagrams illustrating examples of CG models used to explain acquisition of inclusion relationships. [Diagram 5] 1A and 1B are diagrams used to explain examples of overlapping numbers of voxels and CG models. [Figure 6] FIG. 13 is a diagram illustrating an example of a containment structure tree. [Figure 7] FIG. 2 is a diagram showing an example of a CG model and a viewpoint position. [Figure 8] FIG. 13 is a diagram showing an example of a containment structure tree to which a viewpoint has been added and a display flag. [Figure 9] FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a second embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiment may be configured by appropriately combining parts of each embodiment described below. In the following embodiments, the same configurations are described with the same reference symbols.

[0011] [First embodiment] In this embodiment, a system that provides a user with XR (cross reality or x-reality) such as VR (virtual reality), MR (mixed reality), and AR (augmented reality) will be described as an example. In this embodiment, a wearable terminal with limited CPU and GPU processing power will be given as an example of application of the information processing device, but it may also be a stationary personal computer or workstation. The wearable terminal may be not only a wearable device such as a smartphone, but also a computer mounted in the same housing as a head mounted display (HMD), and its form is not limited.

[0012] The information processing device of this embodiment has a function of acquiring multiple viewpoint positions assumed when a user uses XR through interactive operation input. Note that the viewpoint position may include a coordinate position in a three-dimensional space in XR (hereinafter, referred to as XR space), as well as a direction (line of sight) and a width of the field of view from the viewpoint position.

[0013] The information processing device of this embodiment improves the processing efficiency of CG drawing by deleting, among the three-dimensional models of computer graphics (CG), models that are unlikely to be in the user's field of view based on these viewpoint positions. However, if a three-dimensional model is deleted based only on the viewpoint position, it may occur that a three-dimensional model that should have been in the field of view has been deleted when the user observes the XR space from a viewpoint other than these viewpoint positions. In particular, in XR such as VR, MR, and AR, the degree of freedom and range of viewpoints that can be taken at runtime is large, so if a three-dimensional model is deleted based only on the viewpoint position, it may occur that a three-dimensional model that should be displayed is not displayed.

[0014] Therefore, the information processing device of this embodiment analyzes the geometric inclusion relationships between multiple three-dimensional models that compose a scene expressed in CG in the XR space, and generates a tree structure (referred to as an inclusion structure tree) that represents the inclusion relationships between these three-dimensional models. Hereinafter, a scene expressed in CG is referred to as a CG scene, and a three-dimensional model is referred to as a CG model. The details of the analysis process of the geometric inclusion relationships between the CG models will be described later. Then, based on the inclusion structure tree and the viewpoint position acquired by the above-mentioned interactive operation input, the information processing device identifies a CG model that cannot be seen from various viewpoint positions that can be used when observing the XR space, and performs a predetermined simplification process on the identified CG model.

[0015] In this embodiment, the predetermined simplification process includes at least one of a deletion process for deleting the above-mentioned specified CG model, a simplification process for simplifying the shape of the specified CG model, and a combination process for combining the specified CG models. The CG models to be combined are CG models that exist in the same coordinate system and can be considered to be combined as a rigid body. In this embodiment, an example including three processes, a deletion process, a simplification process, and a combination process, is given, but each process may be appropriately combined, such as only a deletion process, a deletion process and a simplification process, or a deletion process and a combination process. Details of the deletion process, simplification process, and combination process will be described later. The information processing device of this embodiment performs these simplification processes to reduce the processing load during rendering of a CG scene.

[0016] Fig. 1 is a block diagram showing an example of a functional configuration of an information processing device 100 according to the first embodiment. Each functional configuration in the information processing device 100 shown in Fig. 1 is realized by, for example, a CPU, a GPU, or a circuit.

[0017] 1, the scene storage unit 102 stores scene graph information indicating parent-child relationships between CG models of a group of CG models constituting a CG scene and coordinates. Note that the scene graph information stored in the scene storage unit 102 is information generated in advance.

[0018] The structure analysis unit 103 is a relationship acquisition unit that analyzes the geometric inclusion relationship structure of the scene graph stored in the scene storage unit 102 to determine which CG model exists inside which CG model, acquires the inclusion relationships between the CG models, and holds the inclusion structure tree. The analysis of the geometric inclusion relationships of CG models and the inclusion structure tree will be described in detail later.

[0019] The operation acquisition unit 104 acquires operation information that is expected to be input by the user when using XR through a prior interactive operation input. Then, the operation information acquired by the operation acquisition unit 104 is stored in the information storage unit 105. Here, operation information expected to be input by the user when using XR may include information on the viewpoint position specified by the user, and information specifying a CG model that the user holds or moves in the XR space. In the case of this embodiment, the operation acquisition unit 104 acquires coordinate information of each of a plurality of viewpoint positions in a three-dimensional space through interactive operation input. The input of each coordinate of the group of viewpoints by the user may be, for example, input via a two-dimensional UI (user interface) having a 3D view, or may be input by numerical values. In addition to these, the input of user operation information may be directly specified using a controller, hand gestures, or the like in a three-dimensional space using, for example, a head-mounted display (HMD).

[0020] In addition, a workflow for CG scene content may involve using a stationary computer or the like with abundant computational resources in advance, and then using the content on a wearable device with limited computational resources. When a stationary computer or the like with abundant computational resources is used in this manner, the operation acquisition unit 104 may acquire information such as coordinate information of a viewpoint trajectory that assumes the movement of the viewpoint during XR use, and an operation history of a CG model that assumes the user moves the CG model.

[0021] The processing determination unit 106 determines the simplification processing to be performed on each CG model by the subsequent deletion processing unit 107 to simplification processing unit 109, based on the inclusion structure tree obtained by the analysis processing in the structure analysis unit 103 and the operation information stored in the information storage unit 105. In this embodiment, the simplification processing includes a deletion processing for deleting one of the CG models, a combination processing for combining one of the CG models with another CG model, and a simplification processing for simplifying the shape of one of the CG models, as described above. That is, the processing determination unit 106 specifies the CG model to be deleted, the CG model to be combined, and the CG model whose shape is to be simplified, based on the inclusion structure tree and the operation information. The details of the CG model specification processing will be described later.

[0022] Then, the processing determination unit 106 sends information about the scene graph stored in the scene storage unit 102 and information indicating the CG models specified to be deleted to a deletion processing unit 107. The processing determination unit 106 also sends information about the scene graph and information indicating the CG models specified to be combined to a combination processing unit 108. The processing determination unit 106 also sends information about the scene graph and information indicating the CG models specified to be simplified to a simplification processing unit 109.

[0023] The deletion processing unit 107 performs a deletion process to delete CG models specified by the processing determination unit 106 as those to be deleted from among the CG models in the scene graph. The combination processing unit 108 performs a combination process to combine CG models that are specified by the processing determination unit 106 as those to be combined, among the CG models in the scene graph. The simplification processing unit 109 performs simplification processing on CG models that are identified by the processing determination unit 106 as those to be simplified, among the CG models in the scene graph. In this embodiment, it is assumed that polygon reduction processing is performed as the simplification processing. Note that the processing determination unit 106 also determines whether or not to simplify the combined CG model after the combining processing. Therefore, when a combined CG model is identified as the CG model to be simplified, the simplification processing unit 109 also performs simplification processing on the combined CG model.

[0024] The simplified scene storage unit 110 stores scene graph information of the CG scene reflecting the results of the simplification processes of the above-mentioned deletion process, combination process, and simplification process. The scene storage unit 102 and the simplified scene storage unit 110 may be the same, in which case the CG scene stored in the scene storage unit 102 is overwritten with the information after the simplification process. Then, during actual XR use, an XR space based on this simplified CG scene is provided to the user. That is, during actual XR use, since CG drawing with high processing efficiency is possible using the simplified CG scene, it is possible to provide an XR space in almost real time even on a wearable device with limited CPU and GPU processing capabilities.

[0025] Fig. 2 is a flowchart showing the flow of information processing performed by the information processing device 100 according to this embodiment. When the functional configuration of Fig. 1 in the information processing device 100 according to this embodiment is realized by a CPU, the information processing program according to this embodiment is a program in which the CPU executes the processing shown in the flowchart of Fig. 2. Of the processing of each step in the flowchart of Fig. 2, the processing of step S201 is performed by the structure analysis unit 103, and the processing of steps S202 to S205 is performed by the processing decision unit 106. The processing of step S206 is performed by the deletion processing unit 107, the processing of step S207 by the combination processing unit 108, the processing of step S208 by the simplification processing unit 109, and the processing of step S209 by the simplified scene storage unit 110.

[0026] First, in step S201, the structure analysis unit 103 analyzes the geometric inclusion relationship structure of the CG model based on the scene graph stored in the scene storage unit 102, and acquires (generates) an inclusion structure tree. Note that the inclusion structure tree may be generated corresponding to the entire CG scene, but for example, when an object of the CG model moves in the CG scene, the inclusion relationship may change. For this reason, the structure analysis unit 103 may generate it for each subscene included in the CG scene, that is, in this case, for each subscene for each coordinate system that may move with the movement of the object of the CG model in the CG scene. Alternatively, the structure analysis unit 103 may generate the inclusion structure tree every time the movement position of the CG model (object) moving in the CG scene changes. Note that, when the inclusion structure tree is generated for each subscene or each change in the movement position, the inclusion structure tree is used by switching between subscenes or changing the movement position.

[0027] FIG. 3 is a flowchart showing the flow of a process for acquiring (generating) a containment structure tree in the structure analysis unit 103. As shown in FIG. In step S301, the structural analysis unit 103 converts each CG model in the CG scene being analyzed into voxels. At this time, the inside of the outermost polygon is filled with voxels. For CG models that have openings, the openings are filled appropriately using a predetermined algorithm. The voxels may be expressed as cubes with a set side length, or as voxels based on an octree.

[0028] Next, in step S302, when the voxels of each CG model in the CG scene being analyzed are arranged in three-dimensional space, the structural analysis unit 103 counts the number of overlaps of the CG models for each voxel and links the overlaps to each voxel.

[0029] The process performed by the structure analysis unit 103 in step S302 will be described below with reference to FIGS. Fig. 4 is a two-dimensional view of the CG models included in the CG scene to be analyzed, arranged in a three-dimensional space, viewed from above, and showing the CG models present inside each CG model. In the example of Fig. 4, room A401, pillar 402, room B403, wagon with casters 404, small room 405, spherical object 406, triangular object 407, and diamond-shaped object 408 represent CG models. That is, the CG scene includes room A401 and room B403, and a pillar 402 is placed in room A401 so as to straddle its interior and exterior. Inside room A401, there is a small room 405 and a wagon with casters 404, inside the small room 405, there is a diamond-shaped object 408, and inside the wagon 404, there is a spherical object 406 and a triangular object 407.

[0030] FIG. 5 is a diagram showing each voxel after the voxelization process in step S301 and each overlap number obtained by counting the overlap number of voxels in step S302 for the CG scene illustrated in FIG. 4. As shown in FIG. 5, the room A401 and the room B403, which are the outermost in the three-dimensional space of the CG scene, have no overlapping areas, so the overlap number is 1. The small room 405 and the wagon 404 are present inside the room A401, so the overlap numbers for the small room 405 and the wagon 404 are each 2. Furthermore, the diamond-shaped object 408 is present inside the small room 405, and the spherical object 406 and the triangular object 407 are stored inside the wagon 404, so the overlap numbers for the diamond-shaped object 408, the spherical object 406, and the triangular object 407 are each 3. In addition, in the case of the pillar 402, the overlap number is 2 in the area present inside the room A401, and the overlap number is 1 in the area present outside the room A401.

[0031] Then, the structure analysis unit 103 repeatedly executes the loop process of steps S303 to S308 in order from the region with the overlap number of 1 to the region with the maximum overlap number (3 in the example of FIG. 5). In step S304 in the loop process, the structural analysis unit 103 searches for voxels (areas) with an overlapping number N (N is 1 to a maximum value) that are being processed in the current loop process. In the next step S305, the structural analysis unit 103 determines whether the number of overlaps N is 1 or not (whether N=1 or not), and if the number of overlaps N is 1, the process proceeds to step S306, and if the number of overlaps N is not 1, that is, if the number of overlaps N is 2 or more, the process proceeds to step S307.

[0032] When proceeding to step S306, the structure analysis unit 103 performs a process of connecting the CG model to which the voxel with the overlap number N of 1 belongs as a node to the inclusion structure tree. Here, the CG model to which the voxel with the overlap number N of 1 belongs is a CG model that is partially or entirely located on the outermost side in the three-dimensional space of the CG scene. Therefore, when proceeding to step S306, the structure analysis unit 103 sets the CG model to which the voxel with the overlap number N of 1 belongs as a node that is directly connected to the root node of the inclusion structure tree.

[0033] On the other hand, if the process proceeds to step S307, the structure analysis unit 103 performs processing to add, as a node to the inclusion structure tree, a CG model having a voxel whose overlap number N is not 1 (where the overlap number N is 2 or more). At this time, the structure analysis unit 103 adds, as a parent node, a CG model whose overlap number N is one less (N-1) among all CG models sharing a voxel whose overlap number N is not 1, and adds, as a child node, a CG model that has not been registered as a parent node to the inclusion structure tree.

[0034] Fig. 6 is a diagram showing an example of a containment structure tree generated by the processing of the flowchart in Fig. 3 for the example CG scenes shown in Fig. 4 and Fig. 5. In the case of the CG scenes shown in Fig. 4 and Fig. 5, in the containment structure tree, as shown in Fig. 6, the nodes for room A and room B are directly connected to the root node, and the nodes for the pillars that exist both inside and outside room A become child nodes with both room A and the root node as parent nodes. In addition, the nodes for the cart, small room, and pillar become child nodes with room A as the parent node and are sibling nodes having a sibling relationship, and the nodes for the spherical object and triangular object become child nodes (sibling nodes) with the cart as the parent node. The node for the diamond-shaped object becomes a child node with the small room as the parent node.

[0035] In this embodiment, as described above, an example is given in which voxels are used when determining the inclusion relationship of each CG model, but the use of voxels is not limited to this. For example, since an inclusion structure tree can be constructed by clarifying the inside / outside relationship between CG models, a bounding box or convex hull may be generated for the CG model, and the inside / outside may be determined approximately using this information. Furthermore, a more accurate inside / outside determination may be performed by combining the interference determination on a polygon basis with the inside / outside determination using the bounding box or convex hull. In addition, when a CG model is provided as a solid model, the inside / outside determination may be performed by Boolean calculation or the like without using voxels.

[0036] Returning to the flowchart of FIG. After the above-mentioned step S201, the process proceeds to step S202, where the processing determination unit 106 identifies a group of viewpoints input by the user based on the operation information acquired by the operation acquisition unit 104 and stored in the information storage unit 105, and adds the group of viewpoints to the containment structure tree. For example, the processing determination unit 106 adds each viewpoint of the group of viewpoints as a node on the containment structure tree, based on the coordinate values ​​of each viewpoint of the group of viewpoints, using rules similar to those described in the flowchart of Fig. 3.

[0037] FIG. 7 shows an example in which a viewpoint a 701, a viewpoint b 702, and a viewpoint c 703 are input as the viewpoints of the group of viewpoints for the CG scene shown in FIG. 8 shows the inclusion structure tree after the viewpoints a701, b702, and c703 shown in FIG. 7 are added as nodes to the inclusion structure tree of FIG. 6 generated based on the examples of FIG. 4 and FIG. 5. Here, the viewpoints a701 and b702 shown in FIG. 7 are present in the room A401, so the overlap number is 2. Therefore, in the inclusion structure tree of FIG. 8, the viewpoints a and b are added as child nodes (sibling nodes) with the room A as the parent node, and further as nodes having a sibling relationship with the pillar. Moreover, the viewpoint c703 shown in FIG. 7 is present in the small room 405, so the overlap number is 3. Therefore, in the inclusion structure tree of FIG. 8, the line of sight c is added as a child node with the small room as the parent node, and as a node having a sibling relationship with the diamond-shaped object.

[0038] Next, in step S203, the process determination unit 106 sets a predetermined flag for a CG model that may be in the field of view in the XR space when XR is used, based on the relationship between the nodes of the inclusion structure tree shown in FIG. 8. In this embodiment, the CG model for which the predetermined flag is set is a CG model that may be in the field of view in the XR space, in other words, a CG model that may be displayed when XR is used, so the predetermined flag is hereinafter referred to as a display flag. In this embodiment, the process determination unit 106 sets a display flag for the CG model of a node that is a sibling of the viewpoint node, the CG model of a node that is a parent node of the viewpoint node, and a CG model that exists within the CG model of the parent node. That is, in the case of the inclusion structure tree of FIG. 8, a display flag is set for the room A that is the parent node of the viewpoints a and b, the wagon, the small room, and the pillar that are sibling nodes of the viewpoints a and b, the small room that is the parent node of the viewpoint c, and the diamond-shaped object that is a sibling node of the viewpoint c.

[0039] For example, if a CG model has an opening inside it and another CG model exists inside that CG model, the CG model inside that CG model with the opening may be visible from the outside through that opening. However, if a display flag is set for a CG model with an opening, but a display flag is not set for a child node in the containment structure tree that has the CG model with the opening as its parent node, the CG model of that child node is deleted or hidden. In other words, in this case, the CG model of the child node that would otherwise be visible through that opening is deleted or hidden.

[0040] In order to deal with such a case, the process determining unit 106 performs the process of step S204 by combining one or more of the following first process, second process, and third process.

[0041] The first process is a process in which, when the size of an opening in a CG model is equal to or larger than a predetermined threshold, a display flag is set for a CG model that is a child node of the CG model having that opening, indicating that the CG model is to be displayed.

[0042] The second process is a process in which, when a CG model that is a child node of a CG model having an opening exists within a predetermined area from the opening, a display flag is set in the CG model of the child node, indicating that the CG model is a display target. When performing the second process, the process determination unit 106 determines, for example, a cone-shaped area that is within a predetermined angle range with respect to the normal vector of the opening as the predetermined area, and sets a display flag in the CG model of the child node that interferes with this cone-shaped area. Of course, the predetermined area is not limited to the cone-shaped area.

[0043] The third process is a process of setting a plurality of viewpoints that orbit the CG model having an opening, and setting display flags for CG models that are within the fields of view of those viewpoints and that are child nodes of the CG model having the opening. Note that the positions and angles for orbiting the CG model having an opening, and the number of viewpoints may be determined arbitrarily depending on the calculation time and the accuracy to be achieved.

[0044] In the above example, the possibility that a CG model existing inside a CG model having an opening can be seen from the outside through the opening was described, but if the CG model existing inside also has an opening, the outside can be seen from inside the CG model through the opening. That is, a CG model having an opening is set as a parent node, and a CG model of a child node existing inside the CG model of the parent node also has an opening, and a viewpoint node can exist inside the CG model of the child node. In this case, for example, if there is no viewpoint node outside the CG model of the parent node, a display flag is not set for the CG model existing outside the CG model of the parent node. In other words, in this case, the CG model existing outside the CG model of the parent node is deleted or hidden, even though it can be seen from inside the CG model of the child node through the opening and further through the opening of the CG model of the parent node.

[0045] In order to be able to handle such cases as well, the processing decision unit 106 of this embodiment performs the processing performed on the child node of the CG model having an opening in the first processing and the second processing described above on the parent node of the CG model having an opening. That is, as a first process, when the size of the opening of a CG model that is a child node is equal to or larger than a predetermined threshold, process determination unit 106 sets a display flag for the CG model of a parent node that has the CG model as a child node. Furthermore, if an opening of a CG model of a parent node, which has a CG model with an opening as a child node, is within a predetermined area from the opening of the CG model of the child node, the processing decision unit 106 sets a display flag for the CG model of the parent node. In this example as well, the predetermined area can be, for example, a cone-shaped area within a predetermined angle range with respect to the normal vector of the opening of the CG model of the child node. Then, if the opening of the CG model in parent mode interferes with the cone-shaped area, the processing decision unit 106 sets a display flag for the CG model in parent mode.

[0046] After the above-mentioned step S204, the process proceeds to step S205, where the process determination unit 106 specifies CG models to be combined in the combination process in the subsequent combination processing unit 108. In this embodiment, CG models whose positions and orientations may change when using XR are specified in advance on the scene graph, and based on this, the process determination unit 106 distinguishes between CG models whose relative positions and orientations do not change. Furthermore, the process determination unit 106 determines CG models under nodes on the same coordinate system in the scene graph as CG models that may be combined. Here, the process determination unit 106 considers that the relative positions and orientations of CG models under nodes specified as CG models whose positions and orientations may change and other CG models in the scene graphs, among the sub-scene graphs on the same coordinate system that may be combined, do not change. Note that the process determination unit 106 may perform the process of step S205 using the size as well as the position and orientation of the CG models. Then, the processing determination unit 106 sets a combination flag in the subordinate CG model of a node whose position, orientation, and size relative to the CG model of the parent node may change, thereby identifying the subordinate CG model as a combination target.

[0047] Next, in step S206, the deletion processing unit 107 performs a deletion process (or a process of making the CG models non-displayable) for which the above-mentioned display flag is not set.

[0048] Next, in step S207, the merging processing unit 108 performs merging processing of the CG models subordinate to the node for which the merging flag was set in step S205. Specifically, the merging processing unit 108 searches for CG models subordinate to the node for which the merging flag was set, and performs merging for meshes for which the same material or texture is set in the searched CG models.

[0049] Next, in step S208, the simplification processing unit 109 performs simplification processing to reduce the number of polygons of the mesh on the CG model for which the display flag is set and the CG model that was combined in step S207. Note that the algorithm for reducing the polygons may be any widely known method, and is not particularly specified. The number of polygons to be reduced may be changed depending on, for example, the size of the CG model, the distance from the set viewpoint position, whether or not it is a child node of a model that has an opening, the size of the opening, and the like. Also, an upper limit value for the number of polygons in the entire CG scene may be set, and the number of polygons to be reduced for each CG model may be determined so as not to exceed the upper limit value. Note that this step itself may be omitted depending on the purpose and the performance of the information processing device.

[0050] After that, in step S209, the simplified scene storage unit 110 stores information on the scene graph of the CG scene that reflects the results of the simplification processing performed in steps S201 to S208.

[0051] As described above, in the information processing device of the first embodiment, a simplification process is performed to omit models that are not in the field of view when the viewpoint is changed during an XR experience, based on the inclusion structure tree that represents the inclusion relationship between CG models and the information on the viewpoint by the user's operation. In this embodiment, when using XR, an XR space with a CG scene that reflects the results of the simplification process is provided to the user. That is, according to this embodiment, the automation range of the lightweighting work of CG models assuming the use of XR can be expanded, and the user's work time can be shortened, and the occurrence of a situation in which even CG models that may be in the field of view are deleted can be reduced.

[0052] [Second embodiment] In the first embodiment, information for performing the simplification process was generated only from a scene graph composed of a CG model. In the second embodiment, an example is described in which the scope of application of the simplification process can be further expanded by considering the structure of real objects in the real space, assuming application to MR or AR in which a CG model is superimposed on the real space. In other words, when a CG model is superimposed on the real space, a CG model or a part of it that is not displayed due to the occlusion of a real object in the real space can be omitted in advance, so that the CG drawing process during rendering and the occlusion determination process of the real object can be reduced.

[0053] Fig. 9 is a block diagram showing an example of the functional configuration of an information processing device 900 according to the second embodiment. The information processing device 900 according to the second embodiment includes a three-dimensional reconstruction unit 901 and a reconstruction data storage unit 902 in addition to the functional configurations shown in Fig. 1. The other components except for the three-dimensional reconstruction unit 901 and the reconstruction data storage unit 902 are generally similar to the functional configurations described in the first embodiment described above, so the following description will focus on the functional configurations and processing that are different from those in the first embodiment.

[0054] The three-dimensional reconstruction unit 901 acquires a three-dimensional model of the real space when the user experiences AR or MR and a three-dimensional model of a real object existing in the real space. The method of acquiring the three-dimensional model of the real space and the real object is not particularly limited, but a method using a stereo camera or a depth sensor may be used, or a method of Structure from Motion using a monocular camera may be used. In addition, the three-dimensional model of the real space and the real object may be generated based on input information of the structure of the real space manually using, for example, a GUI or a 3DUI. The three-dimensional reconstruction unit 901 reconstructs the three-dimensional shape of the real space and the real object by combining any one or two or more of these methods to acquire a three-dimensional model. Note that the three-dimensional reconstruction unit 901 may appropriately divide the three-dimensional model acquired by the three-dimensional reconstruction process. The division of the three-dimensional model may be manual division using a GUI or a 3DUI, or may be performed by implementing segmentation by machine learning. Then, the information of the three-dimensional model acquired by the three-dimensional reconstruction process in the three-dimensional reconstruction unit 901 is stored in the reconstruction data storage unit 902.

[0055] The structure analysis unit 103 of the second embodiment analyzes the inclusion structure of a CG scene in the same manner as in the first embodiment described above, and generates a scene in which the CG model is integrated with a three-dimensional model of real space stored in the reconstruction data storage unit 902. The structure analysis unit 103 then generates an inclusion structure tree based on the scene in which the three-dimensional model of real space and the CG model are integrated. As a result, the inclusion structure tree in the second embodiment is one in which nodes for real space and real objects have been added to the inclusion structure tree described in the first embodiment described above.

[0056] Here, in order to integrate the CG model of the CG scene and the three-dimensional model of the real space, it is necessary to match the coordinate system of the CG scene with the coordinate system of the real space. For this reason, in the case of this embodiment, a predetermined marker is placed in the real space or a certain feature in the real space is used as a marker, and the marker reflected in the image capturing the real space is defined as the origin coordinate when reconstructing the real space. Then, the structure analysis unit 103 specifies the origin coordinate when reconstructing the real space from the marker in the image capturing the real space, and associates the origin coordinate in the coordinate system of the real space with the origin coordinate in the coordinate system of the CG scene. This allows the coordinate system of the real space to match the coordinate system of the CG scene. Alternatively, the coordinate system of the real space and the coordinate system of the CG scene may be manually matched using an interactive UI.

[0057] The information processing device 900 of the second embodiment performs simplification processing of the CG model according to the rules of the flowchart of FIG. 2 described in the first embodiment for a scene in which a CG model and a three-dimensional model of the real space are integrated as described above. In the case of the second embodiment, in step S203, the processing decision unit 106 does not set a display flag for the three-dimensional model of the real space and the real object. In addition, in the case of the present embodiment, the processing decision unit 106 cancels the setting of the display flag for a CG model that can be viewed only from a place where a viewpoint cannot exist due to the presence of a real object, among the CG models for which the display flag is set in step S203. In addition, when an opening of a CG model faces the direction of a real object and the distance from the opening to the real object is equal to or less than a threshold, in step S204, the processing decision unit 106 cancels the setting of the display flag of a subordinate node of the containment structure tree of the CG model having the opening.

[0058] In the second embodiment, the 3D reconstruction processing is performed by the 3D reconstruction unit 901, but the processing may be performed by an information processing device other than the information processing device that performs the simplification processing of the CG scene. For example, the 3D reconstruction processing may be performed in advance by a separate mobile terminal or the like, and the simplification processing may be performed by a stationary information processing device.

[0059] As described above, in the second embodiment, a containment structure tree is generated based on a scene in which a G model and a three-dimensional model of the real space are integrated. In the second embodiment, the display flag is not set for the three-dimensional model of the real object, and even for a CG model with a display flag set, the display flag is appropriately unset based on the positional relationship with the real object, thereby making it possible to expand the range of simplification processing. According to the second embodiment, it is possible to reduce the occurrence of a situation in which even a CG model that may be in the field of view is deleted, and by expanding the range of simplification processing, it is possible to further reduce the user's work time compared to the first embodiment.

[0060] <Hardware configuration example> FIG. 10 is a diagram showing an example of a hardware configuration in which the information processing apparatus according to the first and second embodiments described above can be implemented. 10, a CPU 1020 performs overall control of each device connected via a bus 1010. The CPU 1020 reads out and executes processing steps and programs stored in a read-only memory (ROM) 1030. An operating system (OS), information processing programs according to the present embodiment, device drivers, etc. are stored in the ROM 1030, temporarily stored in a random access memory (RAM) 1040, and appropriately executed by the CPU 1020. An input I / F 1050 inputs a signal from an external device in a format that can be processed by the information processing device of each embodiment. An output I / F 1060 outputs a signal in a format that can be processed by the external device.

[0061] The above-mentioned functional units according to the first and second embodiments are realized by the CPU 1020 expanding a program stored in the ROM 1030 into the RAM 1040 and executing information processing according to the above-mentioned flowchart. For example, when hardware is configured as an alternative to software processing using the CPU 1020, a calculation unit or a circuit corresponding to the processing of each of the above-mentioned functional units according to the first and second embodiments may be configured. That is, each functional unit in the information processing device according to the above-mentioned embodiments may be entirely implemented by hardware, or may be partially executed by software. When the software is executed, the software is executed using the CPU 1020, ROM 1030, RAM 1040, etc. in FIG. 10.

[0062] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

[0063] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a relationship acquisition means for acquiring a geometric inclusion relationship between three-dimensional models constituting a scene represented by computer graphics; An operation acquisition means for acquiring operation information from a user; a determination means for determining, based on the inclusion relationship and the operation information, a three-dimensional model to be subjected to simplification processing among the three-dimensional models constituting the scene, and a simplification processing to be performed on the three-dimensional model to be processed; a processing means for applying the simplification processing determined by the determining means to the three-dimensional model to be processed; 13. An information processing device comprising: (Configuration 2) The information processing device according to configuration 1, wherein the simplification process is one or more of a deletion process for deleting or hiding a three-dimensional model, a combination process for combining three-dimensional models, and a simplification process for simplifying a three-dimensional shape. (Configuration 3) 3. The information processing device according to configuration 1 or 2, wherein the relationship acquisition means acquires a tree structure in which the three-dimensional models are nodes based on a geometric inclusion relationship between the three-dimensional models constituting the scene. (Configuration 4) the operation information acquired by the operation acquisition means includes information for designating a position of a viewpoint for observing the scene, The information processing device according to configuration 3, wherein the determining means adds a node of the viewpoint to the tree structure based on the position of the viewpoint, and identifies, among the three-dimensional models constituting the scene, a three-dimensional model that may be within the field of view of the viewpoint based on a relationship between the node of the three-dimensional model in the tree structure and the node of the viewpoint. (Configuration 5) The information processing device according to configuration 4, wherein the determination means identifies a three-dimensional model that is a sibling node of the viewpoint node in the tree structure and a three-dimensional model that is a parent node of the viewpoint node in the tree structure as three-dimensional models that may be within the field of view of the viewpoint. (Configuration 6) 5. The information processing apparatus according to configuration 4, wherein the determining means sets a flag indicating the specified three-dimensional model in the tree structure. (Configuration 7) The determining means is a first process of determining a three-dimensional model that is a child node of a three-dimensional model having an opening having a size equal to or larger than a predetermined threshold as a three-dimensional model that may be within the field of view of the viewpoint; a second process of determining, as a three-dimensional model that may be within the field of view of the viewpoint, a three-dimensional model that is present within a predetermined area from an opening of the three-dimensional model and has a child node relationship with the three-dimensional model having the opening; a third process of determining a three-dimensional model that is within the field of view of a viewpoint set at a plurality of positions around the three-dimensional model having the opening and has a child node relationship with the three-dimensional model having the opening as a three-dimensional model that may be within the field of view of the viewpoint; 7. The information processing device according to any one of configurations 4 to 6, characterized in that one or more of the information processing devices are used to identify a three-dimensional model that may be within the field of view of the viewpoint. (Configuration 8) The information processing device according to configuration 7, characterized in that, during the second processing, the determining means determines, among three-dimensional models that are child nodes of the three-dimensional model having the opening, a three-dimensional model that interferes with an area within a predetermined angle range with respect to the normal vector of the opening as a three-dimensional model that may be within the field of view of the viewpoint. (Configuration 9) 9. The information processing device according to any one of configurations 3 to 8, wherein the relationship acquisition means acquires the tree structure for each subscene included in the scene. (Configuration 10) The information processing device according to configuration 9, wherein the sub-scene from which the relationship acquisition means acquires the tree structure is a scene for each coordinate system that can be moved in accordance with the movement of the three-dimensional model within the scene. (Configuration 11) 9. The information processing apparatus according to any one of configurations 3 to 8, wherein the relationship acquisition means acquires the tree structure in response to a change in a moving position of a three-dimensional model moving within the scene. (Configuration 12) The information processing device according to any one of configurations 3 to 11, wherein the relationship acquisition means acquires the tree structure based on a number of voxel overlaps between the three-dimensional models constituting the scene when the three-dimensional models are voxelized and arranged in a three-dimensional space. (Configuration 13) The information processing device described in configuration 12, characterized in that the relationship acquisition means directly connects the three-dimensional model to which the voxel with the overlap number of 1 belongs to the root node of the tree structure, and acquires the tree structure in which, among all three-dimensional models sharing the voxel with the overlap number of 2 or more, the three-dimensional model having the voxel with one less overlap number is set as a parent node, and the three-dimensional models that are not parent nodes are set as child nodes. (Configuration 14) 14. The information processing apparatus according to any one of configurations 1 to 13, wherein the operation acquisition means acquires the operation information through an interactive user interface. (Configuration 15) The information processing device described in any one of configurations 1 to 14, characterized in that the operation acquisition means acquires, as the operation information, one or both of information specifying the position of the viewpoint from which the scene is observed and information specifying a three-dimensional model among the three-dimensional models constituting the scene, at least one of whose position, orientation, and size may be changed. (Configuration 16) The information processing device according to any one of configurations 1 to 15, characterized in that the operation acquisition means acquires, as the operation information, at least one of a trajectory of the viewpoint position of a user who has previously experienced the scene and an operation history of the user with respect to a three-dimensional model. (Configuration 17) The method further includes acquiring a three-dimensional model of a real object in a real space, the relationship acquisition means also acquires a geometric inclusion relationship between the three-dimensional model of the computer graphics and the three-dimensional model of the real object; 17. The information processing device according to any one of configurations 1 to 16, wherein the simplification processing determined by the determination means includes a process of deleting or hiding the three-dimensional model generated by computer graphics based on the three-dimensional model of the real object. (Method 1) a relationship acquisition step of acquiring a geometric inclusion relationship between three-dimensional models constituting a scene represented by computer graphics; an operation acquisition step of acquiring operation information by a user; a determination step of determining, based on the inclusion relationship and the operation information, a three-dimensional model to be subjected to simplification processing among the three-dimensional models constituting the scene, and a simplification processing to be performed on the three-dimensional model to be processed; a processing step of applying the simplification processing determined in the determining step to the three-dimensional model to be processed; 13. An information processing method comprising: (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0064] 101: information processing device, 102: scene storage unit, 103: structure analysis unit, 104: operation acquisition unit, 106: processing decision unit, 107: deletion processing unit, 108: combination processing unit, 109: simplification processing unit, 110: simplified scene storage unit

Claims

1. a relationship acquisition means for acquiring a geometric inclusion relationship between three-dimensional models constituting a scene represented by computer graphics; An operation acquisition means for acquiring operation information from a user; a determination means for determining, based on the inclusion relationship and the operation information, a three-dimensional model to be subjected to simplification processing among the three-dimensional models constituting the scene, and a simplification processing to be performed on the three-dimensional model to be processed; a processing means for applying the simplification processing determined by the determining means to the three-dimensional model to be processed; 13. An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the simplification process is one or more of a deletion process for deleting or hiding a three-dimensional model, a combination process for combining three-dimensional models, and a simplification process for simplifying a three-dimensional shape.

3. 2. The information processing apparatus according to claim 1, wherein said relationship acquisition means acquires a tree structure in which the three-dimensional models are nodes based on a geometric inclusion relationship between the three-dimensional models that configure the scene.

4. the operation information acquired by the operation acquisition means includes information for designating a position of a viewpoint for observing the scene, The information processing device according to claim 3, characterized in that the determination means adds a node of the viewpoint to the tree structure based on the position of the viewpoint, and identifies, among the three-dimensional models constituting the scene, a three-dimensional model that may be within the field of view of the viewpoint based on the relationship between the node of the three-dimensional model and the node of the viewpoint in the tree structure.

5. The information processing device according to claim 4, characterized in that the determination means identifies a three-dimensional model that is a sibling node to the viewpoint node in the tree structure and a three-dimensional model that is a parent node to the viewpoint node in the tree structure as three-dimensional models that may be within the field of view of the viewpoint.

6. 5. The information processing apparatus according to claim 4, wherein the determining means sets a flag indicating the specified three-dimensional model in the tree structure.

7. The determining means is a first process of determining a three-dimensional model that is a child node of a three-dimensional model having an opening having a size equal to or larger than a predetermined threshold as a three-dimensional model that may be within the field of view of the viewpoint; a second process of determining, as a three-dimensional model that may be within the field of view of the viewpoint, a three-dimensional model that exists within a predetermined area from an opening of the three-dimensional model and has a child node relationship with the three-dimensional model having the opening; a third process of determining, as a three-dimensional model that may be within the field of view of a viewpoint that is set at a plurality of positions around the three-dimensional model having the opening and that has a child node relationship with the three-dimensional model having the opening, as a three-dimensional model that may be within the field of view of the viewpoint; The information processing apparatus according to claim 4 , wherein one or more of the three-dimensional models that may be within the field of view of the viewpoint are specified.

8. The information processing device according to claim 7, characterized in that, during the second processing, the determination means determines, among three-dimensional models that are child nodes of the three-dimensional model having the opening, a three-dimensional model that interferes with an area within a predetermined angular range with respect to the normal vector of the opening as a three-dimensional model that may be within the field of view of the viewpoint.

9. 4. The information processing apparatus according to claim 3, wherein the relationship acquisition means acquires the tree structure for each subscene included in the scene.

10. 10. The information processing apparatus according to claim 9, wherein the subscene from which the relationship acquisition means acquires the tree structure is a scene for each coordinate system that can be moved in accordance with the movement of the three-dimensional model within the scene.

11. 4. The information processing apparatus according to claim 3, wherein said relationship acquisition means acquires said tree structure in response to a change in a moving position of a three-dimensional model that moves within said scene.

12. The information processing device according to claim 3, characterized in that the relationship acquisition means acquires the tree structure based on the number of voxel overlaps between the three-dimensional models constituting the scene when the three-dimensional models are converted into voxels and arranged in a three-dimensional space.

13. The information processing device according to claim 12, characterized in that the relationship acquisition means acquires the tree structure in which a three-dimensional model to which a voxel with an overlap number of 1 belongs is directly connected to a root node of the tree structure, and among all three-dimensional models sharing a voxel with an overlap number of 2 or more, a three-dimensional model having a voxel with one less overlap number is set as a parent node, and a three-dimensional model that is not a parent node is set as a child node.

14. 2. The information processing apparatus according to claim 1, wherein the operation acquisition means acquires the operation information through an interactive user interface.

15. The information processing device according to claim 1, characterized in that the operation acquisition means acquires as the operation information one or both of information specifying the position of the viewpoint from which the scene is observed and information specifying a three-dimensional model among the three-dimensional models constituting the scene, the position, orientation, or size of which may be changed.

16. 2 . The information processing apparatus according to claim 1 , wherein the operation acquisition means acquires, as the operation information, at least one of a trajectory of a viewpoint position of a user who has previously experienced the scene and a history of operations of the user on a three-dimensional model.

17. The method further includes acquiring a three-dimensional model of a real object in a real space, the relationship acquisition means also acquires a geometric inclusion relationship between the three-dimensional model of the computer graphics and the three-dimensional model of the real object; 17. The information processing apparatus according to claim 1, wherein the simplification process determined by the determining means includes a process of deleting or hiding the three-dimensional model created by the computer graphics based on the three-dimensional model of the physical object.

18. a relationship acquisition step of acquiring a geometric inclusion relationship between three-dimensional models constituting a scene represented by computer graphics; an operation acquisition step of acquiring operation information by a user; a determination step of determining, based on the inclusion relationship and the operation information, a three-dimensional model to be subjected to a simplification process among the three-dimensional models constituting the scene, and a simplification process to be performed on the three-dimensional model to be subjected to the simplification process; a processing step of applying the simplification processing determined in the determining step to the three-dimensional model to be processed; 13. An information processing method comprising:

19. Computer, a relationship acquisition means for acquiring a geometric inclusion relationship between three-dimensional models constituting a scene represented by computer graphics; An operation acquisition means for acquiring information on an operation performed by a user; a determination means for determining, based on the inclusion relationship and the operation information, a three-dimensional model to be subjected to simplification processing among the three-dimensional models constituting the scene, and a simplification processing to be performed on the three-dimensional model to be processed; a processing means for applying the simplification processing determined by the determining means to the three-dimensional model to be processed; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Images