Information processing device, information processing method, and computer program

Abstract

To provide an information processing device that can reduce recognition errors regarding objects.SOLUTION: An information processing device comprises: arrangement knowledge obtaining means which obtains arrangement knowledge information including type information representing the types of two or more observation objects observed by an observer and positional relationship information between the observation objects; inter-object positional relationship recognition means which recognizes object types and positional relationships in the arrangement knowledge information; observation position attitude recognition means which recognizes an observation position attitude, which is a position attitude of observation at the time when the arrangement knowledge information was generated; and spatial positional relationship calculating means which calculates spatial position information between the two or more observation objects based on the positional relationship and the observation position attitude.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a computer program, and the like. [Background technology]

[0002] Image recognition has made it possible for machines to recognize object characteristics such as the type and position of an object captured in an image. In Patent Document 1, a database is updated to output a bounding box surrounding an object from an image and the object type.

[0003] In addition, in Non-Patent Document 1, a 3D shape model is labeled based on the object type information detected in this way. In Non-Patent Document 2, the relationship between objects included in an image is described in text based on the object type information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2019-517701 [Non-Patent Document 1] “CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction”, Tateno et.al, CVPR2019 [Non-Patent Document 2] “I2T:Image Parsing to Text Description,” IEEE,BZ Yao, X.et.al, IEEE Summary of the Invention [Problem to be solved by the invention]

[0005] However, image recognition can sometimes erroneously recognize object types or positional relationships between objects, and there is also the problem that it takes time and effort to prepare a recognition model for executing such recognition processing.

[0006] In view of the above, an object of the present invention is to provide an information processing device capable of reducing object recognition errors. [Means for solving the problem]

[0007] In the information processing device, a placement knowledge acquisition means for acquiring type information indicating types of two or more observation objects observed by an observer and placement knowledge information including positional relationship information between the observation objects; an object positional relationship recognition means for recognizing object types and positional relationships in the placement knowledge information; an observation position and orientation recognition means for recognizing an observation position and orientation which is an observation position and orientation at a time point when the arrangement knowledge information is generated; a spatial positional relationship calculation means for calculating spatial positional information between two or more of the observation targets based on the positional relationship and the observation position and orientation; The present invention is characterized by comprising: Effect of the Invention

[0008] According to the present invention, it is possible to realize an information processing device capable of reducing recognition errors regarding objects. [Brief description of the drawings]

[0009] [Figure 1] 11A and 11B are diagrams showing a specific example of a process for calculating a spatial positional relationship from a positional relationship between objects included in text data. [Diagram 2] FIG. 1 is a diagram showing an example of a scene in which the information processing device according to the first embodiment is used. [Diagram 3] 1 is a functional block diagram showing an example of the configuration of an information processing system 100 according to a first embodiment. [Figure 4]1 is a diagram illustrating an example of a hardware configuration of an information processing system 100 according to a first embodiment. [Diagram 5] 4 is a flowchart showing an example of an information processing method by the system 100 according to the first embodiment. [Figure 6] 11 is a flowchart showing an example of an information processing method by the system 100 according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0011] <Embodiment 1> Three-dimensional spatial recognition can be used for applications such as recognizing the type of object, determining its location, and counting the number of objects, but it can also be applied to a variety of tasks such as location recognition, obstacle avoidance in autonomous driving, and hazard prediction.

[0012] Object detection from images or 3D shape models can be achieved, for example, by using neural networks to select a region of interest (rectangle) and determine the type of object contained therein. However, object detection to date has been specialized in detecting individual objects.

[0013] In other words, the relationships between objects (especially their positional relationships) could not be utilized. For example, chairs are often found next to desks but not on top of them, and desks and chairs are found indoors but not outdoors. These are common sense for humans, but in many cases these properties have been ignored in computer recognition.

[0014] In this embodiment, the arrangement characteristics refer to characteristics of the relationships between objects, such as the combination patterns and frequency of occurrence of positional and contact relationships between objects in real space, changes in these relationships over time, and the probability of their occurrence.

[0015] Such placement characteristics include object type information and positional relationship information, so creating a recognition method and recognition model that take placement characteristics into account is time-consuming in terms of data generation.

[0016] In this embodiment, object type information and positional relationship information are created by collecting document information. Document information refers to data including sentences such as documents and news articles, and is created for the purpose of being read, viewed, or heard by humans.

[0017] Such document information includes object names and positional relation words and phrases contained in the text, so that object type information and positional relation information can be generated. On the other hand, document information may not include sufficient spatial relation information.

[0018] For example, in a text that expresses the positional relationships between objects, the scenery may be projected onto an observation point. The observation point is the relative position and orientation between the object and the observer (the writer of the text) when observing the object's type and location information. Since three-dimensional positional relationship information is lost during the projection stage, in order to create information on the location characteristics of objects from the text, it is necessary to convert the information into three-dimensional positional relationship information and then create the location information.

[0019] In this embodiment, therefore, spatial positional relationship information is calculated from positional relationship information included in the document information. That is, the observation position and orientation are recognized from the document information, and spatial positional relationship information between objects is calculated from the positional relationship between the recognized observation points.

[0020] Fig. 1 is a diagram showing a specific example of a process for calculating a spatial relationship from the positional relationship between objects contained in text data. In Fig. 1, D100 is text data, D110 (D111 to D113) are the positional relationships between objects collected from the text data, and D120 is the spatial relationship generated from D110.

[0021] In FIG. 1, it can be recognized from the layout relationships D111 and D112 that the meter B and the control panel A included in the text are arranged side by side, but it is not possible to determine whether the meter B and the control panel A are in front of or behind each other.

[0022] Also, including D113, contradictions may occur only with information on the positional relationship. In this embodiment, the observation position and orientation (D126, D127) are recognized taking into account the context and chapter headings of the text. Specifically, in the text data D100, the chapter "Operation Procedure" contains the statement "Go around to the front of the processing machine", and it is recognized that the rest of the text is written on the assumption that the user is facing the processing machine from the front, as in D126.

[0023] In addition, because the "Maintenance" chapter states "going around, ~right side," recognize that the rest of the description is based on the assumption that you are facing the processing machine from the right side, as in D127. Calculate the spatial relationship between the operation panel A and the meter B based on the relative positions of D110, D126, and D127, and the observation position and orientation.

[0024] As shown in Fig. 1 above, in this embodiment, the relationship between observation positions is recognized from the object type information and positional relationship included in the document information, and the spatial positional relationship between the objects is calculated. Also, in this embodiment, input data is generated for an object placement characteristic database that aggregates "object placement relationships" in real space. In particular, in this embodiment, a method for generating object placement characteristic data in which observation point dependency has been removed from a placement relationship having observation point dependency will be described.

[0025] 2 is a diagram showing an example of a usage scene of the information processing device according to the embodiment 1. F100 is a text of an operation manual for a processing device F101, and the processing device F101 has an operation panel F102 and a meter F103. F110 is a tablet as the information processing device of this embodiment, which calculates and displays the spatial positional relationship between objects from the text data.

[0026] An object selection area F111, a calculation result display area F112, and a search result display area F113 are displayed on the display of the tablet F110. The object selection area F111 is an area for selecting and displaying the name of an object whose positional relationship the user wishes to confirm.

[0027] The calculation result display area F112 is an area for displaying the spatial relationship of the object selected by the user in the object selection area F111 using a sentence expression, and the search result display area F113 is an area for displaying a portion of the sentence used in the calculation.

[0028] Fig. 3 is a functional block diagram showing a configuration example of the information processing system 100 according to the embodiment 1. Note that some of the functional blocks shown in Fig. 3 are realized by causing a CPU or the like as a computer included in the information processing system 100 to execute a computer program stored in a memory as a storage medium.

[0029] However, some or all of these may be realized by hardware. For the hardware, a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP) may be used. Furthermore, each functional block shown in Fig. 5 does not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths.

[0030] The information processing system 100 includes an information processing device 1, an object layout characteristic generation means 105, and an object layout characteristic database 106. The information processing device 1 includes a layout knowledge acquisition means 101, an inter-object positional relationship recognition means 102, an observation position / posture recognition means 103, a spatial positional relationship calculation means 104, and the like.

[0031] The layout knowledge acquisition means 101 acquires data including object type information and object positional relationship information based on document information (text data, etc.) (not shown) stored in a storage means (not shown), and supplies the data to an inter-object positional relationship recognition means 102 and an observation position / orientation recognition means 103. That is, the layout knowledge acquisition means 101 acquires layout knowledge information including type information indicating the types of two or more observation targets observed by an observer, and positional relationship information between the observation targets.

[0032] The inter-object positional relationship recognition means 102 recognizes the object type information and the positional relationship information based on the data including the object type information and the positional relationship information supplied from the placement knowledge acquisition means 101. That is, the inter-object positional relationship recognition means 102 recognizes the object type and the positional relationship in the placement knowledge information.

[0033] The recognition result is input to an observation position and orientation recognition means 103 and a spatial positional relationship calculation means 104. The positional relationship recognized by the inter-object positional relationship recognition means 102 in this embodiment is the relative position and relative orientation between a plurality of objects.

[0034] The observation position and orientation recognizing means 103 recognizes the observation position and orientation based on the data input by the placement knowledge acquiring means 101 and the object type information and positional relationship information input by the inter-object positional relationship recognizing means 102. In other words, the observation position and orientation recognizing means 103 recognizes the observation position and orientation, which is the observation position and orientation at the time when the placement knowledge information is generated.

[0035] The recognition result by the observation position and orientation recognizing means 103 is input to the spatial relationship calculation means 104. Here, the observation position and orientation refers to the positional relationship between the observer and the observed object recognized based on the data including the object type information and the positional relationship information supplied from the layout knowledge acquiring means 101.

[0036] The spatial positional relationship calculation means 104 calculates the spatial positional relationship based on the object type information and positional relationship information input from the inter-object positional relationship recognition means 102 and the observation position and orientation input from the observation position and orientation recognition means 103. That is, the spatial positional relationship calculation means 104 calculates spatial position information between two or more observation targets based on the positional relationship and the observation position and orientation.

[0037] The spatial positional relationship is a spatial positional relationship that does not change regardless of the observation position and orientation, and has no observation point dependency. That is, the spatial positional relationship calculation means 104 calculates a positional relationship that does not depend on the observation position and orientation. In addition, the spatial positional relationship calculation means 104 inputs the object positional relationship generated from the object type and the spatial positional relationship to the object arrangement characteristic generation means 105.

[0038] The object placement characteristic generating means 105 generates object placement information consisting of the object type information and the spatial position relationship information input from the spatial position relationship calculating means 104 , and inputs the information to an object placement characteristic database 106 .

[0039] The object layout characteristic database 106 is a database that holds layout characteristics that indicate the positional relationships of a plurality of objects. The layout characteristics are knowledge data that generalize the positional relationships of objects in space.

[0040] The object disposition property database 106 in this embodiment is a pre-trained neural network that is trained to infer unknown or erroneous object property information from surrounding object property information.

[0041] As a pre-trained neural network, for example, the Transformer described in the literature ("Attention is All you Need", Ashish.et.el Neural IPS2017) is used. That is, for example, a pre-trained neural network in which the above Transformer is stacked in 24 layers is used.

[0042] In this embodiment, the number of input dimensions and output dimensions of the Transformer is 512, that is, an encoder network or the like is used that can input a maximum of 512 pieces of object characteristic information and obtain an output of the same number of dimensions, 512.

[0043] Specifically, we use the encoder network described in the literature (Jacob et.al, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv 2018).

[0044] 4 is a diagram showing an example of a hardware configuration of the information processing system 100 according to the embodiment 1. H11 is a CPU as a computer, and the CPU executes a program describing the operation in this embodiment, thereby executing the processing according to this embodiment.

[0045] The CPU also controls various devices connected to the system bus H19. H12 is a ROM that stores the BIOS program and the boot program. H13 is a RAM that is used as the main storage device for the CPU H11.

[0046] H14 is an external memory that stores computer programs to be processed by the information processing system 100. The input unit H15 is for inputting information, etc., via a keyboard, mouse, etc. The display unit H16 outputs the results of calculations, etc., of the information processing system 100 to a display device according to instructions from the CPU H11. The display device may be any type, as long as it is a liquid crystal display device, a projector, an LED indicator, etc.

[0047] H17 is a communication interface that communicates information via a network. The communication interface H17 can be Ethernet, USB, serial communication, wireless communication, or any other type. H18 is an I / O as an input / output interface.

[0048] Fig. 5 is a flowchart showing an example of an information processing method by the system 100 according to embodiment 1. Note that the operation of each step in the flowchart in Fig. 5 is performed sequentially by a CPU or the like as a computer in the information processing system 100 executing a computer program stored in a memory.

[0049] In the information processing method of this embodiment, object types and positional relationships are generated from general text data as described above. First, step S501 is an initialization step, in which the system 100 initializes the system. That is, a program is read from the external memory H14, and the information processing device 1 is made operable.

[0050] In step S501, a word group to be referred to in step S503 for recognizing an object position and step S504 for recognizing an observation position and orientation, which will be described later, and a table showing the positional relationship corresponding to each word are loaded into the RAM H13. After a series of initialization processes are completed, the process proceeds to step S502.

[0051] Step S502 is a placement knowledge acquisition step, in which placement knowledge acquisition means 101 acquires data including object type and positional relationship information from a storage means (not shown).

[0052] Step S503 is an inter-object positional relationship recognition step, in which the inter-object positional relationship recognition means 102 recognizes object type information and positional relationship. In this embodiment, since the placement knowledge acquisition means 101 inputs text data, the inter-object positional relationship recognition means 102 recognizes object type information and positional relationship from the text by morphological analysis.

[0053] In this embodiment, words expressing object type information and positional relationships are recognized from a sentence by morphological analysis in natural language processing (NLP). As a word recognition method, for example, the recognition method described in the document by Mikolov et al. (Efficient Estimation of Word Representation in Vector Space. arXiv:1301.3781.) is used.

[0054] In step S503, after recognizing the word, the inter-object positional relationship recognition means 102 recognizes positional relationship information corresponding to the word from the table read in the initialization step S501.

[0055] Step S504 is an observation position / posture recognition step, in which the observation position / posture recognition means 103 recognizes the position / posture of the observer. The observation position / posture recognition means 103 recognizes the position / posture of the observer by recognizing words describing positional relationships, such as prepositions, from the input text data.

[0056] When performing word recognition in step S504, a search is performed for each word group acquired from the text data, and the positional relationship is recognized by referring to the corresponding positional relationship. Note that the observation position and orientation relationship in this embodiment refers to the relative position and orientation between an object and an observer.

[0057] In addition, the positional relationship information in this embodiment refers to quantitative positional relationship information corresponding to object types and words. For example, from the phrase "to the right of the desk," the words "desk" and "to the right" are recognized, and the positional relationship "100 cm to the right of the desk, based on the observer" is recognized.

[0058] Step S505 is a spatial positional relationship calculation step, in which the spatial positional relationship calculation means 104 calculates the object type information and the spatial positional relationship. In this embodiment, the spatial positional relationship calculation means 104 determines a conversion formula between the coordinates set for each object by using the observer as a reference. Specifically, coefficients R and T of a coordinate conversion formula such as Equation 1 are calculated as the spatial positional relationship.

[0059]

number

[0060] In formula 1, xa, ya, za and xb, yb, zb are the coordinates of object A and object B, respectively, R is the rotation component, and T is the translation component. Based on the quantitative position of the object in a coordinate system based on the observation position and orientation input by the observation position and orientation recognition means 103, the spatial positional relationship calculation means 104 can solve the determinant in formula 1 to obtain R and T, and thereby obtain the spatial positional relationship.

[0061] Furthermore, when a plurality of observation positions and orientations are input, the spatial relationship calculation means 104 obtains the spatial relationship in a coordinate system with any of the observation positions and orientations as the reference.

[0062] Step S 506 is an object placement characteristic generating step, in which the object placement characteristic generating means 105 generates object placement characteristic information from the object type information and spatial position relationship information input from the spatial position relationship calculating means 104 .

[0063] In this embodiment, the object placement characteristic information is made up of object type information and spatial position relationship information, and the object placement characteristic generating means 105 stores the object type information, the spatial position relationship information, and address information for each piece of information in the object placement characteristic database 106 .

[0064] Step S507 is an end determination step, in which it is determined whether or not to end the system. In this embodiment, if the user inputs new object type information and positional relationship information, the process returns to step S502 and continues, and if not, the system is ended.

[0065] As described above, according to the first embodiment, an object placement characteristic database can be constructed by removing the dependency of object placement characteristic information, which is made up of object type information and object position information, on the position and orientation of the observer. As a result, it is possible to reduce recognition errors regarding object types and positional relationships between objects, and to save the effort of preparing a recognition model.

[0066] <Modification> In this embodiment, the arrangement knowledge acquiring means 101 acquires text data, but the data format is not limited as long as the information includes objects and their positional relationships.

[0067] For example, voice data including sentences may be acquired, or sentence data such as text may be combined with voice data. Also, it may be combined with object placement information recognized by image recognition. In other words, the placement knowledge information includes at least one of voice data, sentence data, and image data.

[0068] In this embodiment, the observation position / orientation recognition means 103 recognizes the relative relationship between the observation object and the observer as the observation position / orientation, but the observation position / orientation is not limited to the relative position with respect to the observation object, and may be anything that can determine the observation position / orientation. For example, it may be a relative position / orientation between other observation positions / orientations. Alternatively, the observation position / orientation may be determined by an absolute position / orientation such as a direction or longitude / latitude.

[0069] In this embodiment, the observation position / posture recognition means 103 recognizes the observation position / posture from the context of the text, but it may recognize from information other than the context depending on the input data format. For example, the positional relationship may be recognized by recognizing a change in the observation position from the text structure such as a paragraph or chapter, or the positional relationship between the speaker and the object of observation may be recognized from the speaker's lines by speaker recognition of audio data.

[0070] That is, a measuring means for measuring voice data or the like may be provided, and the observation position and orientation recognizing means may recognize the positional relationship between the observer observing the observation target and the observation target at the time when the placement knowledge information is generated, by the measuring means.

[0071] <Embodiment 2> In the first embodiment, a method for calculating quantitative spatial positional relationship information from document information is described. In the second embodiment, a method for generating object arrangement characteristic data from which observation point dependency has been removed by calculating qualitative spatial positional relationship information is described. In the second embodiment, the qualitative positional relationship information includes positional relationships that are not expressed by numerical values.

[0072] The information processing method in the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the information processing method by the information processing system 100 according to the second embodiment. Note that the operation of each step in the flowchart in Fig. 6 is performed sequentially by a CPU or the like as a computer in the system 100 executing a computer program stored in a memory.

[0073] The flow in Fig. 6 differs from the flow in Fig. 5 in the processing contents in steps S601 and S602. The different processing contents of observation position and orientation recognition step S601 and spatial positional relationship calculation step S602 will be explained. Note that the steps with the same reference numbers as Fig. 5 are the same processing, so their explanation will be omitted.

[0074] Step S601 is an observation position and orientation recognition step of the second embodiment, in which the observation position and orientation recognition means 103 recognizes the position and orientation of the observer. In the second embodiment, the observation position and orientation recognition means 103 recognizes words describing positional relationships from input text data, thereby recognizing an object that is not dependent on the orientation of the observed object, and the relative position and orientation of the object with respect to the observer.

[0075] For example, in a three-dimensional coordinate system based on the observer, the position of an object is recognized as a relative position, such as "x coordinate is negative, y coordinate is positive, z coordinate is positive."

[0076] Step S602 is a spatial positional relationship calculation step in the second embodiment, where the spatial positional relationship calculation means 104 calculates the object type information and the spatial positional relationship. In this embodiment, a translation component T is calculated using, for example, the following formula 2, which is a conversion formula between coordinates converted for each object.

[0077]

number

[0078] In the second embodiment, the translation component T is composed of x, y, and z coordinates, and each coordinate is a constant A or -A defined in the information processing program. The spatial positional relationship calculation means 104 calculates the translation component T as A or -A depending on the coordinates of the observation position.

[0079] Specifically, if the x coordinate is negative, the y coordinate is positive, and the z coordinate is positive, then the x coordinate is calculated as -A, the y coordinate is A, and the z coordinate is A.

[0080] According to the second embodiment described above, it is possible to calculate relative positional relationship information in space and generate object placement characteristic data from which observation point dependency has been removed. That is, the observation position and orientation recognizing means recognizes the observation viewpoint and observation direction, and the relative position and orientation of the observed object at the time when the placement knowledge information is generated, based on the information acquired by the placement knowledge acquiring means.

[0081] Therefore, it is possible to reduce recognition errors regarding object types and positional relationships between objects, and to save the effort of preparing recognition models.

[0082] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. In addition, some of the above-mentioned embodiments may be appropriately combined, including the following combinations.

[0083] (Configuration 1) An information processing device comprising: a placement knowledge acquisition means for acquiring placement knowledge information including type information indicating the type of two or more observation objects observed by an observer and positional relationship information between the observation objects; an object-to-object positional relationship recognition means for recognizing the object type and positional relationship in the placement knowledge information; an observation position and orientation recognition means for recognizing the observation position and orientation, which is the position and orientation of the observation at the time when the placement knowledge information is generated; and a spatial positional relationship calculation means for calculating spatial positional information between two or more observation objects based on the positional relationship and the observation position and orientation.

[0084] (Configuration 2) The information processing device according to configuration 1, wherein the spatial positional relationship calculation means calculates a positional relationship that is not dependent on the observation position and orientation.

[0085] (Configuration 3) The information processing device according to configuration 1 or 2, wherein the placement knowledge information includes at least one of voice data, text data, and image data.

[0086] (Configuration 4) An information processing device described in any one of configurations 1 to 3, characterized in that the observation position and orientation recognition means recognizes the observation viewpoint and observation direction at the time when the placement knowledge information is generated, and the relative position and orientation with respect to the observation object, based on the information acquired by the placement knowledge acquisition means.

[0087] (Configuration 5) An information processing device according to any one of configurations 1 to 4, further comprising a measurement means, wherein the observation position and orientation recognition means recognizes, by the measurement means, a positional relationship between an observer observing the observation target at the time when the placement knowledge information is generated and the observation target.

[0088] (Method) An information processing method comprising: a layout knowledge acquisition step of acquiring layout knowledge information including type information indicating the type of two or more observation objects observed by an observer and positional relationship information between the observation objects; an object-to-object positional relationship recognition step of recognizing the object type and positional relationship in the layout knowledge information; an observation position and orientation recognition step of recognizing the observation position and orientation, which is the observation position and orientation at the time when the layout knowledge information is generated; and a spatial positional relationship calculation step of calculating spatial position information between two or more observation objects based on the positional relationship and the observation position and orientation.

[0089] (Program) A computer program for controlling each means of the information processing device according to any one of configurations 1 to 5 by a computer.

[0090] In order to realize a part or all of the control in the above-mentioned embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0091] 101: Placement knowledge acquisition means 102: Means for recognizing positional relationship between objects 103: Observation position and orientation recognition means 104: spatial positional relationship calculation means 105: Object placement characteristic generation means 106: Object arrangement characteristics database

Claims

1. a placement knowledge acquisition means for acquiring type information indicating types of two or more observation objects observed by an observer and placement knowledge information including positional relationship information between the observation objects; an object positional relationship recognition means for recognizing object types and positional relationships in the placement knowledge information; an observation position and orientation recognition means for recognizing an observation position and orientation which is an observation position and orientation at a time point when the arrangement knowledge information is generated; a spatial positional relationship calculation means for calculating spatial positional information between two or more of the observation targets based on the positional relationship and the observation position and orientation; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the spatial positional relationship calculation means calculates a positional relationship that is not dependent on the observation position and orientation.

3. 2. The information processing apparatus according to claim 1, wherein the placement knowledge information includes at least one of voice data, text data, and image data.

4. 2. The information processing apparatus according to claim 1, wherein the observation position and orientation recognizing means recognizes the observation viewpoint and observation direction at the time when the layout knowledge information is generated, and the relative position and orientation of the observation object based on the information acquired by the layout knowledge acquiring means.

5. Further equipped with a measuring means, 2. The information processing apparatus according to claim 1, wherein said observation position and orientation recognition means recognizes, by said measurement means, a positional relationship between an observer who observes said observation target at the time when said placement knowledge information is generated and said observation target.

6. a layout knowledge acquisition step of acquiring type information indicating types of two or more observation objects observed by an observer and layout knowledge information including positional relationship information between the observation objects; an object positional relationship recognition step of recognizing object types and positional relationships in the placement knowledge information; an observation position and orientation recognition step of recognizing an observation position and orientation at a time point when the placement knowledge information is generated; a spatial positional relationship calculation step of calculating spatial position information between two or more of the observation targets based on the positional relationship and the observation position and orientation.

7. A computer program for controlling each unit of the information processing apparatus according to any one of claims 1 to 5 by a computer.

Images