A device and method for reconstructing a scene recording in a virtual space.

Abstract

To provide a device and method for recording and reconstructing a scene, which can easily extract and reconstruct a desired scene on the basis of semantic information and which is capable of appropriate scene extraction with consideration for object interaction.SOLUTION: A device for recording and reconstructing a scene records and reconstructs a scene in a virtual space, comprises: a geometry recording unit that records geometry information expressing the shape and appearance of objects constituting a scene; a record recording unit that records record information, which is time-series information about events enacted by the objects; a knowledge graph recording unit that records the context of the scene as a knowledge graph; and a scene reconstruction unit that reconstructs a desired scene in the virtual space by identifying a scene to be reconstructed from the context of the scene recorded by the knowledge graph recording unit, and obtaining and reintegrating the geometry information and record information necessary to reconstruct the scene.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a scene recording and reconstruction apparatus and a scene recording and reconstruction method in a virtual space.

Background Art

[0002] Virtual Reality (VR) technology that displays a virtual space composed of virtual objects created by Computer Graphics (CG) is widely used in games, sports, telemedicine, maintenance work, etc. For example, it is used for training and verification of various operations in a virtual space, AI learning, etc.

[0003] Here, for example, when reproducing a past scene in a virtual space to realize experiential training, AI agent training, and verification, it is very difficult to search for and select a scene having a desired feature from a huge number of past scenes. This is because in a common ground that reproduces the same environment as the real world digitally, events in the real space are always reproduced and recorded in the virtual space, so the recorded past scenes become huge in scale. Also, in order to find out when and where a specific scene occurred, it is necessary to check all the recorded scenes.

[0004] Also, in a case where a virtual scene is operated multiple times / parallelly, such as in AI agent training, there is a requirement to reproduce the scene within the minimum necessary and appropriate space and time range.

[0005] Patent Document 1 is a prior art document in this field. Patent Document 1 discloses a game device that displays game images obtained by progressing through the game in response to a request for replay display from the player, and displays a 2D image generated by perspective transformation of a virtual 3D space from a virtual camera onto a virtual screen on a display device as a replay, based on character status, pad information input by the player, initial values ​​used to generate random numbers during the progression of the game, and control information of a virtual camera. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2010-142305 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Patent Document 1 records all the information necessary to reproduce a game scene and reconstructs the scene in a virtual space based on the recorded information, and reconstruction is possible within any time range. However, it does not perform semantic fragmentation of the scene's constituent elements, and although it is possible to extract a specific cut by specifying the time, it is not possible to extract cuts based on semantic information such as events and actions, or to select the elements to be reconstructed. This presents a problem in that it cannot meet the demand for reproducing a scene within the minimum necessary and appropriate spatial and temporal range.

[0008] In view of the above problems, the present invention aims to provide a scene recording and reconstruction device and a scene recording and reconstruction method that can easily extract and reconstruct a desired scene based on semantic information, and that can perform appropriate scene extraction considering the interaction of objects. [Means for solving the problem]

[0009] To give one example, the present invention is a scene recording and reconstruction device for recording and reconstructing a scene in a virtual space, comprising: a geometry recording unit for recording geometric information representing the shape and appearance of objects constituting the scene; a record recording unit for recording record information which is time-series information of events performed by the objects; a knowledge graph recording unit for recording the context of the scene as a knowledge graph; and a scene reconstruction unit that, upon request, identifies a scene to be reconstructed from the scene context recorded in the knowledge graph recording unit, and reconstructs the desired scene in the virtual space by obtaining the geometry information and record information necessary for the reconstruction of the scene from the geometry recording unit and the record recording unit, respectively, and reintegrating them. [Effects of the Invention]

[0010] According to the present invention, a scene recording reconstruction device and a scene recording reconstruction method can be provided that can easily extract and reconstruct desired scenes, and that enable appropriate scene extraction. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram of the scene recording and reconstruction system in the embodiment. [Figure 2] This is a schematic diagram of the knowledge graph stored in the knowledge graph recording unit in the embodiment. [Figure 3] This is property information stored in each node of the spatial layer of the knowledge graph in the embodiment. [Figure 4] This refers to the property information stored in the agent node representing a dynamic object within the agent object layer of the knowledge graph in the example. [Figure 5] This refers to the property information stored in the agent node representing a static object within the agent object layer of the knowledge graph in the example. [Figure 6] This is property information stored in the stay node of the stay event layer in the knowledge graph in the example. [Figure 7] It is property information stored in the nodes of the activity layer of the knowledge graph in the embodiment. [Figure 8] It is a processing flowchart of the scene registration unit in the embodiment. [Figure 9] It is a detailed flowchart of the record addition process for existing elements in FIG. 8. [Figure 10] It is a detailed flowchart of the record creation process for new elements in FIG. 8. [Figure 11] It is a processing flowchart of the scene reconstruction unit in the embodiment. [Figure 12] It is a detailed flowchart of the scene automatic expansion process in FIG. 11. [Figure 13] It is a detailed flowchart of the interactive scene expansion process in FIG. 11. [Figure 14] It is a display example of the user interface unit in the interactive scene expansion process of FIG. 13. [Figure 15] In FIG. 14, it is a display example of the user interface unit updated after selecting candidate graph elements.

Embodiments of the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments will be described by taking a virtual training system for fire extinguishing / rescue operations as an example.

Embodiment

[0013] FIG. 1 is a configuration block diagram of the scene recording and reconstruction system in the present embodiment. In FIG. 1, the scene recording and reconstruction system is configured by connecting a scene recording and reconstruction device 101 and a real space sensing device 102 via a network 103.

[0014] The real - space sensing device 102 measures the scene in the real space, that is, the positions and activities of people and objects, etc., with a plurality of sensors 104, integrates the information from each sensor in the sensor data integration unit 105, and transmits the measurement data to the scene recording and reconstruction device 101 via the sensor data transmission unit 106.

[0015] The scene recording and reconstruction device 101 receives the measurement data from the real - space sensing device 102 in the sensor data receiving unit 107. In the scene registration unit 108, the scene is divided into record information (expressing time - series information such as events and actions), geometry information (expressing shapes and appearances), and a knowledge graph (expressing the context such as the relationship before and after the scene, and the association between geometry and records), and each is stored in the record recording unit 109, the geometry recording unit 110, and the knowledge graph recording unit 111 respectively.

[0016] The scene reconstruction unit 112 receives a request from the user interface unit 113, searches, extracts, and identifies the scene to be reconstructed from the context of the scene recorded in the knowledge graph recording unit 111, and acquires the record information and geometry information required for the reconstruction of the scene from the record recording unit 109 and the geometry recording unit 110 respectively and re - integrates them, thereby reconstructing the scene on the 3D virtual space and providing the rendering result or spatial information to the user interface unit 113.

[0017] Note that as a hardware image, the scene recording and reconstruction device 101 consists of a CPU, which is a general - purpose information - processing device, and a storage unit, and is realized by software processing in which the CPU interprets and executes an operation program for realizing each function. Also, storage units such as the record recording unit 109, the geometry recording unit 110, and the knowledge graph recording unit 111 may be provided on the cloud.

[0018] Figure 2 is an overview diagram of the knowledge graph stored in the knowledge graph recording unit 111 in this embodiment. The knowledge graph consists of nodes corresponding to either individual geometry information or record information, and links that represent their relationships. In Figure 2, a firefighting and rescue scene during a disaster is shown as an example.

[0019] In Figure 2, the knowledge graph consists of a spatial layer 201, a dwell event layer 202, an agent / object layer 203, and an activity layer 204.

[0020] The spatial layer 201 consists of nodes (hereinafter also referred to as elements) such as block nodes 205, building nodes 206, floor nodes 207, and room nodes 208, and corresponds to geometric information (shape and appearance). Each node is associated with a relation 209, which is a link representing inclusion and positional relationships.

[0021] The dwell event layer 202 and the agent / object layer 203 consist of dwell nodes 210, agent nodes 211 representing people, robots, etc., and object nodes 212 representing objects. The nodes of the agent / object layer 203 and the spatial layer 201 are linked directly or via the dwell node 210 by relations 209, thereby representing the location of the nodes in the agent / object layer 203. The dwell node 210 corresponds to record information (representing time-series information such as events and actions). The agent node 211 and object node 212 correspond to geometry information (shape and appearance).

[0022] Activity Layer 204 represents the behavior information of the nodes in Agent Object Layer 203. This includes individual agent actions such as walking and gestures. Action The action node 213 is represented by being associated with the agent node 211 that performed the action via a relation 209. This involves an actor and a target / object, such as in rescue or firefighting. Action Action node 214 is represented by being associated with the actor and the target / object nodes via relation 209. Action nodes 213 and 214 correspond to record information (representing time-series information such as events and actions).

[0023] Figure 3 is a table showing the property information stored in each node of the spatial layer 201 of the knowledge graph in this embodiment. In Figure 3, each node of the spatial layer 201 stores a name 301, a node ID 302 to uniquely identify the node, a class 303 representing the type of node, physical geometry data 304 representing the name or file path of the physical geometry information that represents the physical shape and appearance of the node, and spatial geometry data 305 representing the name or file path of the spatial geometry information that represents the spatial range occupied by the node.

[0024] Figure 4 is a table showing the property information stored in the agent node 211 representing a dynamic object in the agent object layer 203 of the knowledge graph in this embodiment. In Figure 4, a dynamic object refers to an object whose position can change, such as furniture that is not fixed in place. The location of a dynamic object is represented by being associated with a node in the spatial layer 201 via a dwell node 210. In Figure 4, the dynamic object node stores a name 401, a node ID 402 to uniquely identify the node, a class 403 representing the type of node, and geometry data 404 representing the name or file path of the geometry information that represents the physical shape and appearance of the dynamic object.

[0025] Figure 5 is a table showing the property information stored in the object node 212 representing a static object in the agent object layer 203 of the knowledge graph in this embodiment. In Figure 5, a static object refers to an object whose position does not change, such as built-in furniture, walls, or floors. The location of a static object node is represented by being directly associated with a node in the spatial layer 201 via a relation 209. In Figure 5, the static object node stores a name 501, a node ID 502 to uniquely identify the node, a class 503 representing the type of node, geometry data 504 representing the name or file path of the geometry information that represents the physical shape and appearance of the static object, and a position 505 representing the detailed position coordinates of the static object.

[0026] Figure 6 is a table showing the property information stored in the stay node 210 of the stay event layer 202 of the knowledge graph in this embodiment. In Figure 6, the stay node 210 stores a name 601, a node ID 602 ​​to uniquely identify the node, a class 603 representing the type of node, the start time 604 and end time 605 of the stay, a trajectory record 606 representing the name or file path of record information that represents the trajectory during the stay, and a skeleton record 607 representing the name or file path of record information that represents the actions and gestures during the stay.

[0027] Figure 7 is a table showing the property information stored in the nodes of the activity layer 204 of the knowledge graph in this embodiment. In Figure 7, the nodes of the activity layer 204 store a name 701, a node ID 702 to uniquely identify the node, a class 703 representing the type of node, the start time 704 and end time 705 of the activity, a trajectory record 706 representing the name or file path of the record information that represents the trajectory during the activity, and a skeleton record 707 representing the name or file path of the record information that represents the actions and gestures during the activity.

[0028] Figure 8 is a processing flowchart of the scene registration unit 108 in this embodiment. In Figure 8, the scene registration unit 108 receives a recording start instruction from the user interface unit 113 and starts recording the scene (S801).

[0029] After scene recording begins, when measurement data is received from the sensor data receiving unit 107 (S802), a fusion process (S803) is performed to compare and integrate it with elements registered in the knowledge graph recording unit 111, and it is determined whether the measurement data is an update to an existing scene element or corresponds to an unknown scene element (S804). If it is an update to an existing element, a record for that element is added (S805). If it is information about an unknown element, an element is added to the knowledge graph and a new record is created (S806).

[0030] Subsequently, the system checks the operation input from the user interface unit 113 (S807) and confirms whether a scene recording termination instruction has been issued (S808). If an instruction has been issued, the scene recording is terminated; otherwise, the system returns to process S802 and resumes scene recording.

[0031] Figure 9 is a detailed flowchart of the record addition process S805 for existing elements in Figure 8. In Figure 9, first, the knowledge graph recording unit 111 is accessed to identify the element to be updated (S901). Next, the updated record information for that element is written to the record recording unit 109 (S902). Subsequently, the updated scene context information, which links the geometry information and the record information, is written to the knowledge graph recording unit 111 (S903).

[0032] Figure 10 is a detailed flowchart of the record creation process S806 for the new element in Figure 8. In Figure 10, first, the geometry recording unit 110 is accessed to identify the geometry data corresponding to the actor and subject / object of the newly added record (S1001). Next, record information such as trajectories and gestures is written to the record recording unit 109 (S1002). Subsequently, the scene context information linking the geometry information and record information is written to the knowledge graph recording unit 111 (S1003).

[0033] Figure 11 is a processing flowchart of the scene reconstruction unit 112 in this embodiment. In Figure 11, the scene reconstruction unit 112 receives a scene reconstruction instruction from the user interface unit 113 and starts the scene reconstruction process (S1101).

[0034] In process S1101, a query graph representing the characteristics of the scene to be searched is received, and in process S1102, the knowledge graph recording unit 111 is accessed and a subgraph that matches the query graph is identified from the accumulated scene context information (S1102).

[0035] Then, based on the subgraph, the subgraph is expanded to comprehensively include events in the same space and time, and an automatic scene expansion is performed to adjust the spatial and temporal range to be targeted for scene reconstruction (S1103). The expanded subgraph is then presented in the user interface unit 113, and interactive scene expansion is performed (S1104). Finally, it is determined whether the scene to be reconstructed has been finalized (S1105), and if not, the process returns to S1103 and the automatic and interactive expansion are repeated.

[0036] If a scene is determined, record information included in the scene is obtained from the record recording unit 109 (S1106), geometry information included in the scene is obtained from the geometry recording unit 110 (S1107), and this information is combined based on the context information stored in the subgraph (S1108).

[0037] Then, based on the combined scene information, the scene is played back in the virtual space, and the rendering result, spatial information, or processing result in the virtual space is presented to the user interface unit 113 (S1109).

[0038] Figure 12 is a detailed flowchart of the scene automatic expansion process S1103 in Figure 11. In Figure 12, first, if the subgraph (target graph) representing the current scene reconstruction target contains isolated geometry elements or event elements, these are removed from the target graph (S1201).

[0039] Next, it is determined whether the target graph contains one or more nodes from spatial layer 201 (S1202). If it does not, the spatial elements to which the elements of the target graph are associated are identified and added to the target graph (S1203).

[0040] Next, it is determined whether the target graph contains only one spatial element (S1204), and if it contains multiple spatial elements, it is further determined whether each spatial element is adjacent to the others (S1205). If both are No, the spatial element that is the common parent of each spatial element contained in the target graph is identified on the knowledge graph and added to the target graph (S1206).

[0041] Next, static object elements associated with each spatial element in the target graph are added to the target graph (S1207), and similarly, stay events associated with each spatial element that overlap with the candidate time range of the target graph are added to the target graph (S1208). Then, dynamic object elements associated with each stay event are added to the target graph (S1209).

[0042] Next, among the activity elements associated with each agent object element in the target graph, those that overlap with the candidate time range are identified on the knowledge graph and added to the target graph (S1210).

[0043] Then, it is checked whether all activity elements in the target graph are associated with agent objects corresponding to the actors and targets / objects (S1211), and if not, Not linked Identify the agent object element associated with the activity element on the knowledge graph and add it to the target graph (S1212). Then return to process S1210.

[0044] Figure 13 is a detailed flowchart of the interactive scene expansion process S1104 in Figure 11. In Figure 13, first, for the spatial elements included in the subgraph (target graph) representing the current scene reconstruction target, the parent spatial elements and spatially adjacent spatial elements are identified on the knowledge graph, and these spatial elements are added to the graph (candidate graph) representing the candidate elements for scene reconstruction (S1301).

[0045] Next, event elements that are temporally adjacent to or near the event (stay / activity) elements included in the target graph are identified on the knowledge graph and added to the candidate graph (S1302). Furthermore, geometric elements (spatial elements, agent / object elements) associated with the added event elements are identified on the knowledge graph and added to the candidate graph (S1303).

[0046] Next, the time range covered by the event elements in the current target graph (candidate time range) is calculated, and the target graph, candidate graph, and candidate time range are presented to the user interface unit 113 (S1304), and the user's candidate selection input is accepted (S1305).

[0047] Then, the content of the input is determined (S1306), and if it is a confirmed input for the target graph, the process is terminated. If it is not a confirmed input, but an input to select an element to be added to the target graph from among the candidate graphs, the selected element is added to the target graph (S1307), and the candidate time range is updated (S1308).

[0048] Figure 14 shows an example of the display on the user interface unit 113 during the presentation process to the user interface unit 113 by process S1304 in the interactive scene expansion process of Figure 13. In Figure 14, the scene search result list screen 1401 displays multiple tabs 1402 of candidate scenes that match the query graph. When each candidate scene is selected, the scene reconstruction range selection screen 1403 is displayed.

[0049] The scene reconstruction range selection screen 1403 displays the target / candidate subgraphs for scene reconstruction. The subgraph 1404 (area enclosed by a dotted line) that matches the query graph and the subgraph 1405 (area enclosed by a solid line) that has been expanded by the automatic scene expansion process S1103 based on the subgraph 1404 that matches the query graph are shown as the subgraphs to be reconstructed. In other words, the expanded subgraph 1405 is a subgraph that has been expanded within the same spatial and temporal range. Furthermore, candidate graph elements 1406 added by the interactive scene expansion process S1104 are displayed. In addition, a time window 1407 representing the candidate time range and a confirmation button 1408 for confirming the scene are displayed.

[0050] The scene reconstruction range selection screen 1403 accepts one of the following inputs: selecting one of the candidate graph elements 1406, or pressing the confirmation button 1408.

[0051] In other words, as a concrete example, Figure 14 shows a firefighting and rescue scene during a disaster, the query graph is "Rescue activities in a room with a fire," and the expanded subgraph 1405, which is the target of reconstruction after automatic scene expansion, is based on the range that matches the query, and the same time range The entire scene to be reconstructed within the same space is "a scene in which rescue workers are providing medical assistance to an injured person while firefighters are simultaneously extinguishing a fire." In addition, candidate graph elements 1406 that are spatially and temporally close to the expanded subgraph 1405 of the current reconstruction target are presented to the user as candidate graphs.

[0052] Figure 15 shows an example of the updated display after selecting one of the candidate graph elements 1406, element 1409, in Figure 14. In Figure 15, the selected element 1409 is added to the target graph, and the subgraph 1505 (area enclosed by a solid line) is updated as the subgraph to be reconstructed to include the selected element 1409 by the automatic scene expansion process S1103 and the interactive scene expansion process S1104. The candidate graph 1506 and the time window 1407 indicating the target time range are also updated.

[0053] In other words, as a concrete example, in Figure 15, the subgraph 1505 to be reconstructed after automatic expansion based on user selection comprehensively represents "a scene in which rescue workers who have completed rescue in the adjacent room B join room A, provide rescue to the injured, and at the same time, firefighters extinguish the fire."

[0054] As described above, this embodiment divides and records a scene into geometry, records, and context, and manipulates the scope of scene reconstruction on a knowledge graph that records the context. That is, the context is represented by a knowledge graph, and the geometry and records are fragmented and stored at the node level of the graph, and reconstruction at the node level is possible as needed. Furthermore, because the knowledge graph maintains the time-series context of the scene, context-aware searches are possible.

[0055] Furthermore, because the geometry and records are fragmented at the node level of the graph, the scope of scene reconstruction can be manipulated on the graph. In addition, scene expansion that takes into account the spatial and temporal proximity and inclusion relationships represented on the knowledge graph can be achieved. That is, based on rules, the scene is automatically expanded to satisfy the consistency and comprehensiveness of the scene, and spatially and temporally adjacent or nearby elements are presented as candidates, and interactive scene expansion is performed to realize the desired scene in combination with automatic expansion. In other words, the scope is automatically expanded and expansion candidates are presented, taking into account the spatial and temporal relationships recorded in the knowledge graph.

[0056] This allows the scope of scene reconstruction to be manipulated on the knowledge graph, making it easy to extract and reconstruct desired scenes based on semantic information (events, actions, etc.). Furthermore, by using the spatial and temporal proximity and inclusion relationships recorded in the knowledge graph, it becomes possible to extract appropriate (comprehensive) scenes that take into account the interactions of objects. In other words, it eliminates concerns that selecting only a portion of the scope might generate misleading scenes that do not adequately represent the interactions of people, objects, etc.

[0057] Therefore, according to this embodiment, a scene recording reconstruction device and scene recording reconstruction method can be provided that can easily extract and reconstruct desired scenes and enable appropriate scene extraction.

[0058] As illustrated above, the present invention allows for easy selection, reproduction, experience, and verification of desired scenes from accumulated examples and past real-world cases. Therefore, scenes focusing on specific individuals or actions can be easily reconstructed, improving the efficiency of training, experience, and verification. Accordingly, the present invention contributes to achieving a high level of economic productivity through technological advancement and innovation, particularly in relation to SDG 8, "Decent Work and Economic Growth," for realizing the Sustainable Development Goals (SDGs).

[0059] Furthermore, the present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. [Explanation of Symbols]

[0060] 101: Scene recording and reconstruction device, 102: Real-world spatial sensing device, 103: Network, 104: Sensor, 105: Sensor data integration unit, 106: Sensor data transmission unit, 107: Sensor data reception unit, 108: Scene registration unit, 109: Record recording unit, 110: Geometry recording unit, 111: Knowledge graph recording unit, 112: Scene reconstruction unit, 113: User interface unit, 201: Spatial layer, 202: Stay event layer, 203: Agent / object layer, 204: Activity layer 205: Block node, 206: Building node, 207: Floor node, 208: Room node, 209: Relation, 210: Stay node, 211: Agent node, 212: Object node, 213, 214: Action node, 1401: Scene search results list screen, 1402: Tabs, 1403: Scene reconstruction range selection screen, 1404: Subgraph matching query graph, 1405: Expanded subgraph, 1406: Candidate graph elements, 1407: Time window, 1408: Confirm button, 1505: Subgraph

Claims

1. A scene recording and reconstruction device for recording and reconstructing scenes in a virtual space, A geometry recording unit that records geometric information representing the shape and appearance of objects that make up the scene, A record recording unit that records record information, which is time-series information of events performed by an object, A knowledge graph recording unit records a knowledge graph composed of nodes corresponding to either individual geometry information or record information of a scene, and links that represent the context, which is the relationship between them. Upon request, the system identifies a subgraph which is the scene to be reconstructed, consisting of nodes corresponding to the geometry information recorded in the knowledge graph recording unit, nodes corresponding to the record information, and links representing the context that associates them. The aforementioned identified subgraph is extended to include all objects and events within the same space and time range. A scene recording reconstruction device characterized by comprising a scene reconstruction unit that reads and integrates the geometry information and the record information from the geometry recording unit and the record recording unit, respectively, based on an extended subgraph, and reconstructs a desired scene in a virtual space.

2. A scene recording reconstruction device according to claim 1, The scene recording and reconstruction device is characterized in that the scene reconstruction unit presents to the user as extension candidates objects and events that are spatially and temporally adjacent or close to the identified subgraph, and allows the user to interactively select the range of scene reconstruction based on the selection and exclusion of candidates.

3. A scene recording and reconstruction method for a scene recording and reconstruction device that records and reconstructs scenes in a virtual space. There is, Geometry information representing the shape and appearance of the objects that make up the scene is recorded in the geometry recording unit. Record information, which is time-series information of events performed by an object, is recorded in the record recording unit. A knowledge graph is recorded in the knowledge graph recording unit, consisting of nodes corresponding to either individual geometry information or record information of the scene, and links that represent the context, which is the relationship between them. Upon request, the system identifies a subgraph which is the scene to be reconstructed, consisting of nodes corresponding to the geometry information recorded in the knowledge graph recording unit, nodes corresponding to the record information, and links representing the context that associates them. The aforementioned identified subgraph is extended to include all objects and events within the same space and time range. A scene recording reconstruction method characterized by reading and integrating the geometry information and the record information from the geometry recording unit and the record recording unit, respectively, based on an extended subgraph, and reconstructing a desired scene in a virtual space.

4. A scene recording reconstruction method according to claim 3, A scene recording reconstruction method characterized by presenting to the user, as extension candidates, objects and events that are spatially and temporally adjacent or close to the identified subgraph, and allowing the user to interactively select the reconstruction range of the scene based on the selection and exclusion of candidates.

Images