Arrangement device, arrangement method, and program

Abstract

To more efficiently realize estimation of relationship between a position attitude of an object and the position attitude of a CG model.SOLUTION: An arrangement device arranges a CG model in which a shape is substantially coincides with an object so as to overlap with the object in a specific space. The arrangement device comprises: position acquisition means which acquires 3-dimensional positions of a plurality of markers arranged in a first object; selection means which selects a first CG model in which a shape is substantially coincides with the first object, based on the 3-dimensional positions of the plurality of markers; estimation means which estimates relationship of a relative position attitude between the first CG model selected by the selection means and the first object, based on the 3-dimensional positions of the plurality of markers; and arrangement means which arranges the first CG model so as to overlap with the first object in the specific space, based on the relationship of the position attitude estimated by the estimation means.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an apparatus, method, and program for arranging a CG model. [Background technology]

[0002] Conventionally, systems that render CG (Computer Graphics) models in response to the movement of a display device superimpose the CG model onto an object held by the user or an object moving in real space. In such systems, it is important to control the CG model to an appropriate position and orientation according to the position and orientation of the object.

[0003] Patent Document 1 describes a technique in which a range sensor is mounted on a system and the relationship in position and orientation between an object and a CG model is calculated based on a range image obtained from the range sensor and the depth value of the CG model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Pat. No. 1,081,7724 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, the user needs to select a CG model that corresponds to an object, and there are cases in which it is difficult to efficiently estimate the relationship between the position and posture (position and posture) of an object and that of a CG model.

[0006] Therefore, an object of the present invention is to more efficiently estimate the relationship between the position and orientation of an object and that of a CG model. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: An arrangement device that arranges a CG model having a shape that is approximately identical to an object so as to overlap the object in a specific space, a position acquisition means for acquiring three-dimensional positions of a plurality of markers arranged on a first object; a selection means for selecting a first CG model having a shape substantially identical to that of the first object based on the three-dimensional positions of the plurality of markers; an estimation means for estimating a relative position and orientation relationship between the first CG model selected by the selection means and the first object based on three-dimensional positions of the plurality of markers; a placement means for placing the first CG model so as to overlap the first object in the specific space based on the position and orientation relationship estimated by the estimation means; The placement device is characterized by having:

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: A method for arranging a CG model having a shape substantially identical to that of an object in a specific space so as to overlap the object, comprising the steps of: a position acquisition step of acquiring three-dimensional positions of a plurality of markers arranged on a first object; a selection step of selecting a first CG model having a shape substantially identical to that of the first object based on the three-dimensional positions of the plurality of markers; an estimation step of estimating a relative position and orientation relationship between the first CG model selected in the selection step and the first object based on the three-dimensional positions of the plurality of markers; and, a placement step of placing the first CG model in the specific space so as to overlap with the first object, based on the position and orientation relationship estimated in the estimation step; The method is characterized by comprising the steps of: Effect of the Invention

[0009] According to the present invention, it is possible to more efficiently estimate the relationship between the position and orientation of an object and that of a CG model. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of an HMD according to a first embodiment. [Diagram 2] 3 is a diagram showing the internal configuration of an alignment estimation unit according to the first embodiment. FIG. [Diagram 3] 4 is a flowchart of a display process of a composite image according to the first embodiment. [Figure 4] 4 is a flowchart of a process of an alignment estimation unit according to the first embodiment. [Diagram 5] FIG. 2 is a diagram illustrating a mixed reality space according to the first embodiment. [Figure 6] FIG. 2 is a hardware configuration diagram of the HMD according to the first embodiment. [Figure 7] FIG. 11 is a diagram showing the internal configuration of an alignment estimation unit according to the second embodiment. [Figure 8] 10 is a flowchart of a composite image display process according to the second embodiment. [Figure 9] 10 is a flowchart of a process of an alignment estimation unit according to the second embodiment. [Figure 10] FIG. 11 is a block diagram of an HMD according to a third embodiment. [Figure 11] FIG. 4 is a diagram for explaining group ID association according to the first embodiment. [Figure 12] FIG. 2 is a diagram illustrating a rectangular marker according to the first embodiment. [Figure 13] 4A to 4C are diagrams for explaining estimation of alignment information according to the first embodiment. [Figure 14] FIG. 4 is a diagram illustrating marker information and feature point information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following, an object that is placed in real space and that can be moved by a user is referred to as a "movable object."

[0012] <Embodiment 1> FIG. 5 shows a mixed reality space (MR space) in which a user 590 experiences MR. FIG. 5 is a schematic diagram showing the relationship between a user 590 wearing an HMD 1, a tool 520, a pedestal 550, a virtual tool 500, a virtual part 560, and a virtual pedestal 530 on which the virtual part 560 is placed. In the mixed reality space, there are objects arranged in the real space and virtual objects (CG models) that are virtually expressed. The user 590, the tool 520, and the pedestal 550 are objects that are actually arranged in the real space. The virtual tool 500, the virtual pedestal 530, and the virtual part 560 are virtual objects (CG models). The virtual pedestal 530 and the virtual part 560 are objects that are linked to each other.

[0013] Virtual tool 500 is disposed so as to overlap tool 520. Virtual pedestal 530 is disposed so as to overlap pedestal 550. By looking at display unit 110 of HMD 1 worn on the head, user 590 can see an image in which virtual tool 500, virtual pedestal 530, and virtual part 560 are combined with a captured image (an image of real space captured by imaging unit 103).

[0014] In the first embodiment, as shown in FIG. 5, a plurality of markers 510 (markers 510A to 510E) used for seamlessly combining a virtual object with an object in real space are arranged (fixed) on a tool 520 and a base 550.

[0015] The marker 510 is, for example, a rectangular index having an identification ID of the ArUco method, the detection function of which is implemented in the OpenCV library. Note that in the first embodiment, the shape of the marker 510 is not limited to being rectangular. As long as the position of the center point of the marker 510 can be measured based on a plurality of images acquired by the imaging unit 103, the shape of the marker 510 may be any shape.

[0016] (HMD configuration) 6 is a hardware configuration diagram of the HMD 1 according to embodiment 1. The HMD 1 includes a CPU 601, a ROM 602, an external interface 603, a storage medium drive 605, an external storage device 606, a RAM 607, a mouse 608, a keyboard 609, and a system bus 610. The HMD 1 also includes an imaging unit 103, a display unit 110, and an operation unit 120.

[0017] A CPU (Central Processing Unit) 601 is a control unit that controls the entire HMD 1.

[0018] The ROM 602 is a read only memory that stores programs and parameters that do not require modification. The ROM 602 is a memory only. A predetermined information processing program is stored in the ROM 602 as a program code readable by the CPU 601. The CPU 601 executes the processing indicated by this program code.

[0019] The external interface 603 transmits and receives image signals and files to and from the imaging unit 103 .

[0020] The storage media drive 605 is a device that permanently stores data. The storage media drive 605 records data on a physical medium and reads recorded data.

[0021] The external storage device 606 is an external storage device installed inside the HMD 1. Alternatively, the external storage device 606 is an external storage device that is detachable from the HMD 1. The external storage device 606 includes a hard disk, a flash memory, a floppy disk (FD), an optical disk (such as a Compact Disk (CD)), a magnetic disk, an optical card, an IC card, a memory card, or the like. Video files acquired by the imaging unit 103, and the like, are written to the external storage device 606.

[0022] The RAM 607 is a random access memory (RAM) that temporarily stores programs and data supplied from an external device or the like.

[0023] The mouse 608 and keyboard 609 are operation members that allow the user to give instructions to the HMD 1 .

[0024] A system bus 610 is a system bus that connects each component so that they can communicate with each other.

[0025] The imaging unit 103 is, for example, a camera that captures an image (hereinafter, referred to as a "camera image") by capturing an image of a scene in real space. The imaging unit 103 may be a color camera or a monochrome camera. The imaging unit 103 may also be a stereo camera for displaying a stereo image on the HMD 1. In the first embodiment, the imaging unit 103 is a stereo camera in order to place a display panel near both eyes of the user 590. The imaging unit 103 outputs the camera image to the model placement device 100 described below.

[0026] The display unit 110 is a device that displays an image. In the first embodiment, the display unit 110 uses a display mounted on the HMD 1. Alternatively, an external monitor disposed outside the HMD 1 may be used instead. The display unit 110 may display the same image on the display of the HMD 1 and the external monitor at the same time.

[0027] The operation unit 120 is a device provided for the user 590 to give instructions. For example, the operation unit 120 may be a button. The operation unit 120 may include a mouse 608 or a keyboard 609.

[0028] 1 shows a block diagram of an HMD 1 worn on the head of an experiencer 590. The HMD 1 includes a model placement device 100, an imaging unit 103, a display unit 110, and an operation unit 120.

[0029] The model placement device 100 generates an image representing a composite space in which a CG model is placed in a real space. The model placement device 100 is an information processing device such as a personal computer, a smartphone, or a server on the cloud. The model placement device 100 is connected to an imaging unit 103, a display unit 110, and an operation unit 120. For this reason, the model placement device 100 has a hardware configuration other than the "imaging unit 103, the display unit 110, and the operation unit 120" in the HMD 1 shown in FIG. 6.

[0030] (Configuration of model placement device) 1 shows the functional configuration of a model placement device 100. The model placement device 100 has an image acquisition unit 105, an operation acquisition unit 125, a storage unit 150, a marker management unit 155, a synthesis unit 170, an alignment estimation unit 180, a camera estimation unit 190, and an object estimation unit 195.

[0031] The image acquisition unit 105 acquires the camera image from the imaging unit 103 and stores the camera image in the storage unit 150 .

[0032] The operation acquisition unit 125 outputs a signal indicating the operation received by the operation unit 120 to the storage unit 150 or the alignment estimation unit 180.

[0033] The storage unit 150 is configured with a RAM 607. Therefore, the storage unit 150 temporarily holds data stored in the ROM 602 and data generated by each component. The storage unit 150 manages data required to realize the arrangement of a CG model. For example, the storage unit 150 holds camera internal parameters, geometric information of the CG model, alignment information of the CG model, camera images, virtual images, information on the camera position and orientation (position and orientation), information on the position and orientation of movable objects, and marker information.

[0034] The camera internal parameters include information such as the focal length and principal point of the imaging unit 103. The geometric information of the CG model includes edge information of the CG model, a texture image of the CG model, and vertex positions of the CG model (three-dimensional polygon) in a coordinate system unique to the CG model (hereinafter referred to as a "model coordinate system"). The alignment information of the CG model is information indicating the relationship between the position and orientation of the CG model and a movable object on which the CG model is to be superimposed. More specifically, the alignment information is information of a matrix (4×4 matrix) for converting the position and orientation of the CG model in the world coordinate system to the position and orientation of the movable object when the CG model overlaps with the movable object in the mixed reality space. The alignment information is information necessary for aligning the CG model with the movable object since the shapes and sizes of the CG model and the movable object do not completely match each other.

[0035] The virtual image is an image obtained by rendering a CG model. The camera position and orientation information is a parameter that represents the position and orientation (position and orientation) of the imaging unit 108 in the world coordinate system. The movable object position and orientation information is a parameter that represents the position and orientation of the movable object in the world coordinate system.

[0036] The marker information is information about each marker. As shown in FIG. 14A, the marker information holds a marker ID, marker arrangement information, group ID, and marker detection information. The marker ID is a numerical value for identifying each marker. The marker arrangement information is information about a three-dimensional position (X, Y, Z) that defines the position of each marker. The marker arrangement information may be set in advance or may be automatically generated during the MR experience. The marker arrangement information may also include posture information of the marker. The group ID is a numerical value that indicates the movable object on which the marker is arranged. The group ID is a numerical value that is uniquely assigned to each movable object. The group ID is attribute information of each marker. The group ID may be set in advance or may be automatically generated during the MR experience. The marker detection information includes two-dimensional coordinates of the marker in the camera image, and information about the position and posture of the imaging unit 103 at the time of detecting the marker (at the time of imaging the marker).

[0037] In the first embodiment, the experiencer 590 sets a different group ID for each movable object in advance. In addition, the experiencer 590 assigns to each marker a group ID corresponding to the movable object on which the marker is placed. In the first embodiment, the marker placement information indicates the three-dimensional position (X, Y, Z) of the marker in a reference coordinate system. Here, one of the multiple markers placed on one movable object is defined as a reference marker, and the "reference coordinate system" is a coordinate system that has the position of the reference marker as its origin.

[0038] The marker management unit 155 stores marker information of a plurality of markers arranged in real space in the storage unit 150. In the first embodiment, the marker management unit 155 acquires marker arrangement information and a group ID of each marker according to settings by the experiencer 590. For this reason, the marker management unit 155 operates as a position acquisition unit that acquires the three-dimensional position of each marker.

[0039] Furthermore, based on a camera image in which the marker appears, the marker management unit 155 detects the two-dimensional coordinates of the marker in the camera image and the position and orientation of the imaging unit 103 when the marker is detected (when the marker is imaged). Then, the marker management unit 155 stores information on the two-dimensional coordinates of the marker in the camera image and information on the position and orientation of the imaging unit 103 when the marker is detected in the storage unit 150 as marker detection information.

[0040] The virtual image generating unit 160 renders the CG model based on the information on the position and orientation of the camera, the information on the position and orientation of the movable object, the camera internal parameters, the geometric information of the CG model, the alignment information, and the like. In the first embodiment, the virtual image generating unit 160 acquires the information on the position and orientation of the CG model in the world coordinate system based on the information on the position and orientation of the movable object and the alignment information. The virtual image generating unit 160 renders the CG model in the coordinate system of the camera image based on the information on the position and orientation of the CG model in the world coordinate system and the information on the position and orientation of the camera. The virtual image generating unit 160 stores the rendered image of the CG model in the storage unit 150 as a virtual image. As a result, a virtual image is generated in which the CG model overlaps the movable object when the virtual image and the camera image are combined.

[0041] The synthesis unit 170 synthesizes the camera image and the virtual image to generate a synthetic image. In the synthetic image (mixed reality space represented by the synthetic image), the CG model is superimposed on the movable object. In other words, the synthesis unit 170 also functions as a placement unit that places the CG model in the mixed reality space so as to be superimposed on the movable object. The synthetic image is displayed on the display unit 110.

[0042] The alignment estimation unit 180 estimates (determines) alignment information such that the CG model overlaps the movable object in the composite image based on the geometric information and marker information of the CG model. The internal configuration of the alignment estimation unit 180 in the first embodiment will be described later with reference to FIG. The alignment information determined by the alignment estimation unit 180 is stored in the storage unit 150.

[0043] The camera estimation unit 190 estimates the position and orientation of the imaging unit 103 when capturing a camera image (the position and orientation of the imaging unit 103 in the world coordinate system). In the first embodiment, for example, the position and orientation of the imaging unit 103 may be estimated based on markers not placed on a movable object (for example, markers 510F to 510H in FIG. 5) among markers captured in the camera image. The camera estimation unit 190 may estimate the position and orientation of the imaging unit 103 based on marker placement information set in advance and two-dimensional coordinates of the markers in the camera image. The camera estimation unit 190 may use a method of estimating the self-position and orientation of the camera, such as SLAM, instead of estimating the position and orientation of the imaging unit 103 based on the markers.

[0044] The object estimation unit 195 estimates the position and orientation of a movable object when a camera image is captured. The position and orientation of the movable object are estimated based on a plurality of markers arranged on the movable object. For example, the object estimation unit 195 may estimate the position and orientation of the movable object in a reference coordinate system with a reference marker as the origin.

[0045] An example of a specific process of the object estimation unit 195 according to the first embodiment will be described with reference to Fig. 5. For example, the object estimation unit 195 identifies a marker that is assigned either the group ID of the movable object 520 or the group ID of the movable object 550 among markers captured in a camera image. In the example shown in Fig. 5, markers 510A to 510C are arranged on the movable object 520. Markers 510D to 510E are arranged on the movable object 550.

[0046] The object estimation unit 195 estimates the positions of the markers 510A to 510C in the camera coordinate system based on, for example, the marker information (positions in the reference coordinate system) of the markers 510A to 510C to which the group ID of the movable object 520 is assigned and the camera image. The camera coordinate system is a coordinate system based on the position and orientation of the imaging unit 108. The object estimation unit 195 converts the positions of the markers 510A to 510C in the camera coordinate system into coordinate positions according to the position of the imaging unit 103 (position in the world coordinate system) estimated by the camera estimation unit 190. This allows the object estimation unit 195 to estimate the positions of the markers 510A to 510C in the world coordinate system. The object estimation unit 195 estimates the position and orientation of the movable object 520 in the world coordinate system based on the positions of the markers 510A to 510C in the world coordinate system. The position of the movable object 520 is, for example, the center position of the reference marker or the center of gravity position of the movable object 520. The pose of the movable object 520 is, for example, the orientation from marker 510A to marker 510C.

[0047] (Internal configuration of alignment estimation unit) 2 is a block diagram showing the internal configuration of the alignment estimation unit 180. By referring to a plurality of markers arranged on a movable object, the alignment estimation unit 180 selects a CG model having a shape that substantially matches the shape of the movable object from among a plurality of CG models stored in the storage unit 150. Then, the alignment estimation unit 180 associates the movable object with the selected CG model. This makes it possible to omit manual work (the work of associating the movable object with the CG model), thereby shortening the preparation time for the MR experience.

[0048] The alignment estimation section 180 has a volume calculation section 210, a shape calculation section 220, a group ID setting section 240, and an estimation section 260. Below, each component of the alignment estimation section 180 will be described for an example in which alignment information for one CG model is determined.

[0049] The volume calculation unit 210 obtains information on all vertex positions (hereinafter referred to as "vertex position group") of the CG model (a three-dimensional polygon representing the CG model) from the geometric information of the CG model. The extraction unit 210 obtains a rectangular parallelepiped with the smallest volume that contains the group of vertex positions. The volume calculation unit 210 also obtains the volume of that rectangular parallelepiped.

[0050] For example, the volume calculation unit 210 calculates a rectangular parallelepiped with a side parallel to the X-axis, a side parallel to the Y-axis, and a side parallel to the Z-axis of the model coordinate system, and with the minimum volume that contains the vertex positions. Next, the volume calculation unit 210 calculates the minimum rectangular parallelepiped (a rectangular parallelepiped with a side parallel to the X-axis, a side parallel to the Y-axis, and a side parallel to the Z-axis of the model coordinate system) that contains the vertex positions of the CG model after rotating the vertex positions of the CG model by +10 degrees around the X-axis as the rotation axis. The volume calculation unit 210 repeats this +10-degree rotation and calculation of the rectangular parallelepiped until the rotation angle of the X-axis reaches +350 degrees, and calculates a total of 36 patterns of rectangular parallelepipeds.

[0051] Furthermore, the volume calculation unit 210 rotates the vertex positions of the CG model by +10 degrees each time around the Y axis, similar to the rotation about the X axis, to calculate the 35 patterns of the smallest rectangular parallelepipeds. The volume calculation unit 210 rotates the vertex positions of the CG model by +10 degrees each time around the Z axis, similar to the rotation about the X axis, to calculate the 35 patterns of the smallest rectangular parallelepipeds. Note that these rectangular parallelepipeds have sides parallel to the X axis, Y axis, and Z axis of the model coordinate system.

[0052] The volume calculation unit 210 selects the smallest rectangular parallelepiped (hereinafter referred to as "CG rectangular parallelepiped") from the 106 patterns of calculated rectangular parallelepipeds. Then, the volume calculation unit 210 calculates the volume of the CG rectangular parallelepiped and sets the volume of the CG rectangular parallelepiped as the volume of the CG model. When multiple CG models are stored in the storage unit 150, the volume calculation unit 210 calculates the volumes of the multiple CG models by the same method as described above. Note that any method may be used to calculate the volume of the CG model.

[0053] Furthermore, the volume calculation unit 210 detects the three-dimensional positions of the multiple markers belonging to one group ID based on the marker arrangement information and the group ID. The volume calculation unit 210 calculates the volume of the smallest rectangular parallelepiped that contains the three-dimensional positions (three-dimensional position group) of the detected multiple markers. The calculation of the volume of this rectangular parallelepiped is performed in the same manner as the calculation of the volume of the CG rectangular parallelepiped. Hereinafter, the "smallest rectangular parallelepiped that contains the multiple markers belonging to one group ID (multiple markers placed on one movable object)" is called the "marker rectangular parallelepiped". Then, the volume calculation unit 210 sets the volume of the marker rectangular parallelepiped as the volume of the movable object of the group ID corresponding to the marker rectangular parallelepiped. In FIG. 12, the marker rectangular parallelepiped 1201 contains all of the four markers 510, 510A, 510B, 510C, and 510J.

[0054] The shape calculation unit 220 calculates the ratio of sides of the CG rectangular parallelepiped (the ratio of sides in the width direction, height direction, and depth direction). Furthermore, the shape calculation unit 220 calculates the ratio of sides of the marker rectangular parallelepiped for each group ID.

[0055] The group ID setting unit 240 associates the group ID of a movable object whose shape is approximately the same as that of the CG model with the CG model. Specifically, the group ID setting unit 240 selects a marker rectangular parallelepiped having a volume closest to the volume of the CG rectangular parallelepiped of the CG model. Then, the group ID setting unit 240 associates the group ID of the marker corresponding to the selected marker rectangular parallelepiped with the CG model.

[0056] For example, as shown in Fig. 11, assume that there are three CG models 500, 530, and 560. In this case, assume that group ID 1 is assigned to markers 510A, 510B, 510C, and 510J, and group ID 2 is assigned to markers 510D, 510E, 510K, and 510L.

[0057] The group ID setting unit 240 calculates the "difference between the volume of the CG rectangular parallelepiped of the CG model 500 and the volume of the marker rectangular parallelepiped of group ID 1" and the "difference between the volume of the CG rectangular parallelepiped of the CG model 500 and the volume of the marker rectangular parallelepiped of group ID 2." Then, the group ID setting unit 240 associates the group ID corresponding to the smaller of the two differences with the CG model 500.

[0058] Here, the group ID setting unit 240 determines that it is impossible to associate the CG model with the group ID when there are multiple "marker rectangular parallelepipeds whose volume difference with the volume of the CG rectangular parallelepiped of the CG model 500 is smaller than a predetermined threshold value." As in the example of FIG. 11, when the volumes of the CG rectangular parallelepipeds of all the CG models stored in the storage unit 150 are significantly different from each other, there is a high possibility that accurate association will be achieved. However, when CG models with a small difference in size are stored, accurate association may not be achieved by simply referring to the volume difference. In such a case, the group ID setting unit 240 attempts to associate the CG model with the group ID based on the information on the side ratio calculated by the shape calculation unit 220.

[0059] The group ID setting unit 240 associates, for example, a group ID corresponding to a marker rectangular parallelepiped having a minimum difference in side ratio (ratio) from the side ratio (ratio) of the CG rectangular parallelepiped with the CG model. The "difference between the side ratio (ratio) of the CG rectangular parallelepiped and the side ratio (ratio) of the marker rectangular parallelepiped" is, for example, the sum of the "difference in side length in the height direction" and the "difference in side length in the depth direction" when the side length in the width direction is normalized to "1". For example, if the side ratio of the CG rectangular parallelepiped is width:height:depth=1:2:3 and the side ratio of the marker rectangular parallelepiped is width:height:depth=5:11:16, the side ratio of the marker rectangular parallelepiped is converted to width:height:depth=1:2.2:3.2. Then, the difference between the two ratios is calculated as (2.2-2)+(3.2-3)=0.4.

[0060] If there is no marker rectangular parallelepiped whose difference in side ratio from that of the CG rectangular parallelepiped is equal to or less than a predetermined value, the group ID setting unit 240 may notify the experiencer 590 that there is no group ID corresponding to the CG model. In this case, the group ID setting unit 240 may prompt the experiencer 590 to input a group ID corresponding to the CG model. The experiencer 590 can input the group ID corresponding to the CG model by operating the operation acquisition unit 125.

[0061] Furthermore, the group ID setting unit 240 may use any method other than those based on the volume or side ratio, as long as it can more appropriately associate a CG model with a group ID (a movable object whose shape is approximately the same as that of the CG model). For example, the similarity between the shapes of two rectangular parallelepipeds may be calculated by any method, and the CG model may be associated with a group ID according to the similarity. For example, the similarity between the shapes of the two rectangular parallelepipeds may be calculated by a method in which an eigenvector is found by singular value decomposition of a group of three-dimensional vertices, and an angle difference between a first eigenvector and a second eigenvector is used as the degree of similarity.

[0062] The estimation unit 260 calculates alignment information for converting the position and orientation of the CG model into the position and orientation of a movable object based on the CG model and the marker information of the group ID associated with the CG model. That is, alignment information representing the relative position and orientation between the CG model and the movable object of the group ID associated with the CG model is determined. The details of the processing by the estimation unit 260 will be described later. The estimation unit 260 stores the alignment information in the storage unit 150.

[0063] (Display processing of composite images) The display process of the composite image according to the first embodiment will be described in detail with reference to the flowchart in Fig. 3. Before the process in the flowchart in Fig. 3 starts, the camera internal parameters are stored in the ROM 602. It is assumed that the database stores information such as the location of the object, the CG model, and marker information. It is also assumed that N movable objects and M markers are placed in the real space.

[0064] In step S 300 , the CPU 601 reads the camera internal parameters from the ROM 602 to the storage unit 150 .

[0065] In step S 310 , the CPU 601 reads out the geometric information of the CG model from the ROM 602 to the storage unit 150 .

[0066] In step S320, the CPU 601 reads the marker information from the ROM 602 to the storage unit 150. Here, the user sets a marker ID, marker placement information, and group ID for each marker in advance. Therefore, the marker information stored in the ROM 602 includes at least the marker ID, marker placement information, and group ID.

[0067] In step S330, CPU 601 controls alignment estimation unit 180 to estimate (determine) alignment information of N CG models corresponding to N movable objects. Alignment estimation unit 180 estimates alignment information of N CG models based on camera internal parameters, geometric information, and marker information read into storage unit 150. Details of the process of step S330 will be described later with reference to the flowchart in FIG.

[0068] In step S340, CPU 601 controls marker management unit 155 to detect M markers from the camera image. Marker management unit 155 associates each of the detected M markers with appropriate marker information based on the marker ID stored in storage unit 150. Marker management unit 155 stores the two-dimensional coordinates of each of the M markers in the camera image in the marker detection information in storage unit 150.

[0069] In step S350, the CPU 601 controls the camera estimation unit 190 to estimate the position and orientation of the imaging unit 103. In the first embodiment, the camera estimation unit 190 estimates the position and orientation of the imaging unit 103 in the world coordinate system based on, for example, "the two-dimensional coordinates in the camera image and the marker arrangement information" of each of a plurality of markers that are not arranged on any of the N movable objects.

[0070] In step S360, the CPU 601 controls the object estimation unit 195 to estimate the positions and orientations of the N movable objects on which the markers are arranged. In the first embodiment, the object estimation unit 195 estimates the positions and orientations of the movable objects in the world coordinate system based on the "three-dimensional positions of the markers arranged on the movable objects (three-dimensional positions indicated by the marker arrangement information)" and the "position and orientation of the imaging unit 103."

[0071] In step S370, CPU 601 controls virtual image generation unit 160 to render N CG models corresponding to the N movable objects. Virtual image generation unit 160 generates a virtual image by rendering the N CG models based on alignment information and the like. At this time, by using the alignment information, the virtual image is generated so that "in a mixed reality space represented by a composite image in which a camera image and a virtual image are composited, each of the N CG models overlaps with a movable object corresponding to that CG model."

[0072] In step S380, the CPU 601 controls the synthesis unit 170 to synthesize the camera image and the virtual image to generate a synthetic image. At this time, in the mixed reality space represented by the synthetic image, each of the N CG models is arranged so as to overlap with a movable object corresponding to the CG model. It has been done.

[0073] In step S390, CPU 601 displays the composite image on display unit 110. In this way, the composite image is presented to experiencer 590, and experiencer 590 can experience the mixed reality space.

[0074] In step S395, the CPU 601 determines whether or not an end instruction has been given from the experiencer 590. If it is determined that an end instruction has been given, the process of this flowchart ends. If it is determined that an end instruction has not been given, the process proceeds to step S340, where the process of this flowchart is executed for a new camera image acquired from the imaging unit 103.

[0075] (Details of step S330) The process of step S330 will be described in detail with reference to the flowchart of Fig. 4. Here, the process of the flowchart of Fig. 4 is executed for each CG model stored in storage unit 150. Hereinafter, one CG model that is the target of the process of the flowchart of Fig. 4 will be referred to as a "target model."

[0076] In step S400, alignment estimation unit 180 determines whether or not alignment information of the target model is stored in storage unit 150. If it is determined that alignment information of the target model is not stored, the process proceeds to step S405. If it is determined that alignment information of the target model is stored, the process of this flowchart ends.

[0077] In step S405, volume calculation unit 210 calculates a CG rectangular parallelepiped of the target model. Volume calculation unit 210 also calculates a marker rectangular parallelepiped for each group ID (i.e., movable object). In the following, it is assumed that there are L group IDs (i.e., movable objects), and L marker rectangular parallelepipeds have been calculated. Volume calculation unit 210 calculates the volume of the CG rectangular parallelepiped of the target model (= the volume of the target model) and the volumes of the L marker rectangular parallelepipeds (= the volumes of the L movable objects).

[0078] In step S410, shape calculation unit 220 calculates the side ratio of the CG rectangular parallelepiped of the target model and the side ratio of the L number of marker rectangular parallelepipeds. Note that the information on the volume and side ratio of the CG rectangular parallelepiped of the target model and the information on the volume and side ratio of the L number of marker rectangular parallelepipeds may be stored in advance in storage unit 150 without executing the processes of steps S405 and S410.

[0079] In step S430, the group ID setting unit 240 selects (determines) a group ID corresponding to the target model based on the volume of the CG rectangular parallelepiped of the target model and the volumes of the L number of marker rectangular parallelepipeds. Alternatively, the group ID setting unit 240 may select (determine) a group ID corresponding to the target model based on the ratio of the sides of the CG rectangular parallelepiped of the target model and the ratio of the sides of the L number of marker rectangular parallelepipeds.

[0080] In step S450, the estimation unit 260 estimates alignment information of the target model based on the correspondence between the target model and the group ID, the geometric information of the CG model, and the marker information. Specifically, the estimation unit 260 selects a movable object corresponding to the target model based on the correspondence between the target model and the group ID. Then, the estimation unit 260 determines alignment information of the target model indicating the relative position and orientation between the target model and the movable object corresponding to the target model based on the geometric information and marker information of the target model.

[0081] In the following, the estimation unit 2 will be described with reference to the case where the CG model 500 shown in FIG. 13A is the target model. The process of 60 will be described. First, the estimation unit 260 selects the movable object 520 as the movable object corresponding to the CG model 500 based on the correspondence between the target model and the group ID. Note that there is a one-to-one correspondence between CG models and movable objects. Therefore, "selecting the movable object 520 as the movable object corresponding to the CG model 500" can be said to be "selecting the CG model 500 as the CG model corresponding to the movable object 520 (a CG model having a shape that approximately matches the shape of the movable object 520)."

[0082] After that, the estimation unit 260 estimates the center of gravity positions of the four markers 510 (markers 510A, 510B, 510C, 510J) belonging to the group ID of the CG model 500. Then, the estimation unit 260 tentatively positions the CG model 500 such that the reference position (e.g., the origin of the reference coordinate system) of the CG model 500 is located at the estimated center of gravity position.

[0083] Next, estimation unit 260 uses the provisionally placed position and orientation of CG model 500 as an initial state, and calculates final alignment information of CG model 500 using the steepest descent method or the like. Fig. 13B shows a schematic diagram of calculation of alignment information of CG model 500. In Fig. 13B, points 1310A, 1310B, 1310C, and 1310J indicate the positions of the polygon surface of CG model 500.

[0084] In the first embodiment, the estimation unit 260 obtains the distance between the positions of the markers 510A, 510B, 510C, and 510J of the movable object 520 and the polygon surface of the CG model 500. The estimation unit 260 calculates the position and orientation of the CG model 500 such that the sum of the distances between the positions of the markers and the polygon surface is minimized by a known steepest descent method. That is, the position and orientation of the CG model 500 is calculated such that the sum of the distance between the marker 510A and the point 1310A, the distance between the marker 510B and the point 1310B, the distance between the marker 510C and the point 1310C, and the distance between the marker 510J and the point 1310J is minimized. After that, the estimation unit 260 determines information of a matrix that converts the calculated position and orientation of the CG model 500 in the world coordinate system into the position and orientation of the movable object 520 in the world coordinate system as the alignment information of the CG model 500.

[0085] The distance between the position of each marker and the polygon surface of the CG model can be calculated by the following process. First, the estimation unit 260 selects a vertex position that is close to the marker position from among the vertex positions of the CG model in the initial state. Next, the estimation unit 260 obtains an equation for a polygon plane that is close to the marker position from edge information including the selected vertex position. Then, the estimation unit 260 calculates the distance between the marker position and the polygon plane using a known method.

[0086] When using the steepest descent method, if the initial state is not close to the correct value, the calculation may diverge and the optimal solution may not be calculated. Therefore, the initial position of the CG model is set, for example, so that the center of gravity of the positions of multiple markers belonging to a specific group ID matches the center of gravity of the vertex position of the CG model. The initial posture of the CG model may be set according to the rotation angle at which the distance is minimized by rotating the posture of the CG model by 0 degrees, 90 degrees, 180 degrees, and 270 degrees on the X-axis, Y-axis, and Z-axis, respectively, and calculating the above-mentioned distance for each posture.

[0087] In this manner, the process of the flowchart in FIG. 4 is executed the same number of times as the number of CG models corresponding to movable objects, thereby estimating alignment information for all of the CG models.

[0088] 4 may be executed for each group ID. In this case, one group ID that is the target of the processing in the flowchart of FIG. 4 is called a "target ID." In this case, the processing in step S400 is not executed. Then, in step S405, the volume calculation unit 210 calculates the volume of the CG rectangular parallelepipeds of all the CG models stored in the storage unit 150. and calculates the volume of the marker rectangular parallelepiped of the movable object to which the target ID is assigned. In step S410, the shape calculation unit 220 calculates the ratio of the sides of the CG rectangular parallelepiped of all CG models stored in the storage unit 150 and the ratio of the sides of the marker rectangular parallelepiped. Then, in step S430, the group ID setting unit 240 selects a CG model corresponding to the target ID (a CG model having a shape that is approximately the same as the shape of the movable object of the target ID) from all CG models stored in the storage unit 150 according to the volume or side ratio of the rectangular parallelepiped. The group ID setting unit 240 associates the selected CG model with the target ID. In step S450, if the alignment information of the CG model selected by the group ID setting unit 240 is not stored in the storage unit 150, the alignment information of the CG model is determined.

[0089] According to the first embodiment, the model placement device that places the CG model so as to overlap the movable object associates the movable object with the CG model based on the CG model and marker information, and estimates alignment information of the CG model. This makes it possible to more efficiently estimate the relationship between the position and posture of the movable object and the position and posture of the CG model. This also makes it possible to reduce the effort and time required for the experiencer 590 (user) to prepare in advance for the MR experience.

[0090] <Embodiment 2> In the first embodiment, the group ID and the marker arrangement information in the marker information are set in advance by the experiencer 590. That is, the experiencer 590 sets (calibrates) in advance the positional relationship between the multiple markers arranged (fixed) on the movable object and the correspondence between the movable object and the multiple markers arranged on the movable object.

[0091] On the other hand, the second embodiment does not assume that the placement information of the marker information and the group ID are set in advance by the experiencer 590. In the second embodiment, the model placement device 100 sets the group ID and the marker placement information, regardless of the setting by the experiencer 590 (user). This makes it possible to further shorten the preparation time of the experiencer 590 in advance, and reduce the burden on the experiencer 590.

[0092] The configuration of the model placement device 100 according to the second embodiment is similar to that of the model placement device 100 according to the first embodiment described with reference to Figures 1 and 2. However, the processing of the marker management unit 155 and the alignment estimation unit 180 is different from that of the first embodiment.

[0093] In a mode for calculating the marker placement (hereinafter referred to as the "placement calculation mode"), the marker management unit 155 generates (obtains) marker placement information based on marker detection information indicating the results of detecting markers from a plurality of camera images.

[0094] The storage unit 150 stores information on each marker placed in real space detected from a camera image as marker detection information. The marker management unit 155 acquires two-dimensional coordinates of the marker in each of the multiple camera images and information on the position and orientation of the camera when the marker was detected. The marker management unit 155 executes a known bundle adjustment based on the two-dimensional coordinates of the marker and the camera position and orientation when the marker was detected, as well as on information on the camera's internal parameters. As a result of the bundle adjustment, the three-dimensional position of each marker (three-dimensional position in the reference coordinate system) is estimated. The information on the three-dimensional position of the marker is stored in the storage unit 150 as "marker placement information" in the marker information.

[0095] Here, in the bundle adjustment, it is assumed that the markers attached to the movable objects are stationary. For this reason, the display unit 110 may warn the experiencer 590 in advance by displaying a message requesting "Do not move the movable objects in the placement calculation mode." In addition, once the bundle adjustment process is completed, the display unit 110 may notify the experiencer 590 that the movable objects can be moved.

[0096] In addition, in the placement calculation mode, a marker placed on the bottom surface of a movable object or a marker hidden behind a movable object may not be detected from the camera image. When there is a marker that has not been detected in the placement calculation mode, the user 590 may instruct the model placement device 100 to recalculate the marker placement via the operation unit 120. When such an instruction is given, the model placement device 100 may transition to the placement calculation mode again. Then, the marker management unit 155 may perform bundle adjustment based on a camera image acquired again by the imaging unit 103 that has moved to a position where the marker can be detected.

[0097] The alignment estimation unit 180 calculates a "marker group ID" that has not yet been specified before executing the process described in the embodiment 1. Fig. 7 shows an internal configuration of the alignment estimation unit 180 according to the embodiment 2. The alignment estimation unit 180 has a group generation unit 700 in addition to a volume calculation unit 210, a shape calculation unit 220, a group ID setting unit 240, and an estimation unit 260.

[0098] The group generation unit 700 detects movable objects that have been moved individually by the experiencer 590. The group generation unit 700 determines that all markers that have moved together with the movable object are markers placed on the movable object, and sets (assigns) a new identical group ID (group ID of the moved movable object) to all markers that have moved together with the movable object. Details of the group ID setting process will be described later with reference to the flowchart of FIG. 9. When the experiencer 590 moves a movable object, the experiencer 590 notifies the model arrangement device 100 of the fact that the movable object has been moved via the operation acquisition unit 125.

[0099] A composite image display process according to the second embodiment will be described with reference to the flowchart in Fig. 8. In the process of the flowchart in Fig. 8, the steps in which the same processes as those in the first embodiment are executed are given the same numbers, and therefore the description of those steps will be omitted. In the flowchart in Fig. 8, an alignment estimation process (step S800) is executed after a position and orientation estimation process (step S360) of each movable object.

[0100] In step S800, CPU 601 controls alignment estimation unit 180 to estimate alignment information of the CG model based on the information stored in storage unit 150. Details of the process of step S800 will be described with reference to the flowchart of FIG.

[0101] 9, step S900 is added before step S400, as compared to the flowchart of FIG 4 according to embodiment 1. Note that while the processes of steps 400 to S450 are executed for each CG model, the process of step S900 is not executed for each CG model.

[0102] In step S900, when the group generation unit 700 receives an event that a movable object has moved from the operation acquisition unit 125, it compares the current positions of all markers with the previous positions of all markers. Here, since the marker arrangement information is calculated every time a movable object moves, the group generation unit 700 can calculate the current positions of all markers and the previous positions of all markers based on the previous marker arrangement information and the current marker arrangement information. Then, the group generation unit 700 assigns the same group ID (group ID of the moved movable object) to all markers whose positions have changed.

[0103] Here, when multiple movable objects are arranged in the real space, the user 590 moves the movable objects one by one individually, and each time, uses the operation unit 120 to notify the model arrangement device 100 of an event. Each time an event is received from the operation acquisition unit 125, the group generation unit 700 calculates the current positions of all markers and the previous positions of all markers, and calculates the difference in positions as follows: The same group ID is assigned to all markers with a group ID greater than 0 (or a predetermined difference). Once group IDs have been assigned to all markers placed on all movable objects, the process proceeds to step S400.

[0104] According to the second embodiment, the experiencer 590 does not need to assign a group ID to a marker, and the experiencer 590 only needs to move a movable object and notify the model placement device 100 of the movement of the movable object. This reduces the processing burden on the experiencer 590.

[0105] <Embodiment 3> In the first and second embodiments, the model placement device 100 estimates alignment information of the CG model based on the marker information. In the third embodiment, the model placement device 100 estimates alignment information based on one or more feature points (such as a pattern captured in a camera image) present on the movable object.

[0106] Even if the number of markers arranged on the movable object is small, the pattern of the movable object, like the marker, can represent the geometric information of the movable object and can be treated as information for estimating the surface shape. In other words, the presence of the pattern of the movable object increases the information for estimating the shape and alignment information of the CG model. Therefore, even if it is necessary to select a group ID corresponding to the CG model from a large number of group IDs, the accuracy of matching the CG model with the group ID can be improved. In addition, in the process of estimating the alignment information (the process of step S450), the accuracy of estimating the alignment information can be improved by increasing the number of samples of the position where the distance to the CG model is calculated.

[0107] 10 shows functional blocks of the HMD 1 according to embodiment 3. The HMD 1 according to embodiment 3 includes a feature information detection unit 1100 in addition to the configuration of the HMD 1 according to embodiment 2.

[0108] The feature information detection unit 1100 detects feature points of a movable object captured in a camera image stored in the storage unit 150. The feature information detection unit 1100 associates a "feature point ID" with the feature point and stores the feature point information in the storage unit 150.

[0109] As shown in FIG. 14B, the feature point information includes a feature point ID, feature point arrangement information, a group ID, and feature point detection information. The feature point ID is a numerical value for identifying each feature point. The feature point arrangement information is information on a three-dimensional position (X, Y, Z) in a reference coordinate system that indicates the position of each feature point. The group ID is an ID that is common to the group ID held by the marker information. The group ID is attribute information of each feature point. The feature point detection information includes information on the two-dimensional coordinates of the feature point in the camera image, the number of times the feature point was detected, and the position and orientation of the imaging unit 103 when the feature point was detected.

[0110] In the third embodiment, the group generation unit 700 assigns group IDs to feature points in the same manner as to marker information. For example, in response to a notification of an event in which a movable object has moved, the group generation unit 700 may assign a new identical group ID (the group ID of the moved movable object) to all markers and feature points whose positions have changed since immediately before.

[0111] The display process of the composite image according to the third embodiment is similar to the display process of the composite image according to the second embodiment. However, some of the processes in steps S900, S405, and S450 are different from those in the third embodiment.

[0112] In step S900, the group generation unit 700 assigns group IDs to the multiple markers and feature points placed on the movable object.

[0113] In step S405, the volume calculation unit 210 sets a marker cuboid that contains the three-dimensional positions of all markers belonging to the same group ID and the three-dimensional positions of all feature points, and calculates the volume of the marker cuboid (volume of the movable object). At this time, if the number of markers arranged on one movable object is small, when the marker cuboid is calculated as in the first embodiment, the difference in volume between the marker cuboid and the CG cuboid becomes large, and there is a possibility that the correspondence will not be performed correctly. On the other hand, in the third embodiment, in addition to the three-dimensional positions of the markers, the three-dimensional positions of the feature points are also contained in the marker cuboid, so that it is possible to obtain an effect that the difference in volume between the marker cuboid and the CG cuboid to be associated is reduced. This improves the possibility of correctly associating the CG model with the group ID.

[0114] In step S450, the estimation unit 260 may calculate not only the distance between the marker positions and the surface of the CG model, but also the distance between the feature points and the surface of the CG model. Then, the estimation unit 260 estimates alignment information indicating the relative position and orientation relationship between the CG model and the movable object such that the sum of the distances between the "positions of all markers and the positions of all feature points" and the "surface of the CG model" is minimized.

[0115] However, the positions of the detected feature points may include errors due to image processing recognition, and may be deviated from the position of the surface of the movable object. For this reason, the alignment estimation unit 180 may use only those feature points whose detection counts (detection counts from camera images) stored in the feature point detection information exceed a predetermined number of times.

[0116] According to the third embodiment, since the alignment information can be estimated based on not only the markers but also the feature points of the movable object, the alignment information can be estimated more accurately. In addition, by using the feature points of the movable object, it is possible to accurately associate the CG model with the group ID.

[0117] In the above-mentioned embodiments, an HMD (video see-through type HMD) that displays a composite image representing a mixed reality space by synthesizing a virtual image with a captured image has been described. However, when the HMD is an optical see-through type that allows the real space to be viewed through the display surface, the HMD only needs to display a virtual image on the display surface. Even in this case, the user can view a mixed reality space in which a CG model is placed in the real space. That is, even in this case, the HMD can place the CG model so that it overlaps with a movable object in the mixed reality space.

[0118] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0119] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0120] Each functional unit in each of the above embodiments (variations) may be implemented as separate hardware. Alternatively, the functions of two or more functional units may be realized by common hardware. Each of the functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may be realized by hardware such as an ASIC, an FPGA, or a DSP, or may not be realized by such hardware. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0121] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0122] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An arrangement device that arranges a CG model having a shape that is approximately identical to an object so as to overlap the object in a specific space, a position acquisition means for acquiring three-dimensional positions of a plurality of markers arranged on a first object; a selection means for selecting a first CG model having a shape substantially identical to that of the first object based on the three-dimensional positions of the plurality of markers; an estimation means for estimating a relative position and orientation relationship between the first CG model selected by the selection means and the first object based on three-dimensional positions of the plurality of markers; a placement means for placing the first CG model so as to overlap the first object in the specific space based on the position and orientation relationship estimated by the estimation means; A placement device comprising: (Configuration 2) the selection means selects the first CG model from among a plurality of CG models based on the three-dimensional positions of the plurality of markers. 2. The arrangement device according to configuration 1, (Configuration 3) the selection means selects the first CG model based on a volume of the first object and volumes of the plurality of CG models. 3. The arrangement device according to configuration 2. (Configuration 4) The arrangement device described in configuration 3, wherein the selection means 1) calculates, based on the three-dimensional positions of the plurality of markers, a volume of the smallest rectangular parallelepiped that contains the plurality of markers as a volume of the first object, and 2) calculates, as a volume of the CG model, a volume of the smallest rectangular parallelepiped that contains each vertex of the CG model. (Configuration 5) the selection means selects the first CG model based on a ratio of sides of a smallest rectangular parallelepiped that contains the plurality of markers and a ratio of sides of a smallest rectangular parallelepiped that contains each vertex of the CG model. 5. An arrangement device according to any one of configurations 2 to 4. (Configuration 6) the estimation means estimates a relative position and orientation relationship between the first CG model and the first object such that a sum of distances between a polygonal surface of the first CG model and each of a plurality of markers arranged on the first object is minimum; 6. An arrangement device according to any one of configurations 1 to 5. (Configuration 7) The position acquisition means acquires three-dimensional positions of the plurality of markers based on an image of the first object captured by an imaging device. 7. An arrangement device according to any one of configurations 1 to 6. (Configuration 8) The arrangement device according to configuration 7, further comprising a determination means for determining, when the first object moves, all markers that move with the first object as the plurality of markers arranged on the first object. (Configuration 9) the position acquisition means acquires three-dimensional positions of the plurality of markers and three-dimensional positions of one or a plurality of feature points on the first object based on an image of the first object captured by an imaging device; the estimation means estimates a relative position and orientation relationship between the first CG model selected by the selection means and the first object, based on the three-dimensional positions of the plurality of markers and the three-dimensional positions of the one or more feature points; 9. An arrangement device according to any one of configurations 1 to 8. (Configuration 10) the position acquisition means acquires three-dimensional positions of the plurality of markers and three-dimensional positions of one or a plurality of feature points on the first object based on a plurality of images captured by the imaging device of the first object; the estimation means estimates a relative position and orientation relationship between the first CG model selected by the selection means and the first object, based on a three-dimensional position of a feature point that has been detected a predetermined number of times among the one or more feature points and the three-dimensional positions of the multiple markers; 10. The arrangement device according to configuration 9. (method) A method for arranging a CG model having a shape substantially identical to that of an object in a specific space so as to overlap the object, comprising the steps of: a position acquisition step of acquiring three-dimensional positions of a plurality of markers arranged on a first object; a selection step of selecting a first CG model having a shape substantially identical to that of the first object based on the three-dimensional positions of the plurality of markers; an estimation step of estimating a relative position and orientation relationship between the first CG model selected in the selection step and the first object based on three-dimensional positions of the plurality of markers; a placement step of placing the first CG model in the specific space so as to overlap with the first object, based on the position and orientation relationship estimated in the estimation step; 13. An arrangement method comprising: (program) A program for causing a computer to function as each of the means of the arrangement device according to any one of configurations 1 to 10. [Explanation of symbols]

[0123] 100: model placement device, 170: synthesis unit, 180: alignment estimation unit, 195: object estimation unit, 240: Group ID setting unit, 260: Estimation unit

Claims

1. 1. An arrangement device that arranges a CG model having a shape that is approximately identical to that of an object so as to overlap the object in a specific space, comprising: a position acquisition means for acquiring three-dimensional positions of a plurality of markers arranged on a first object; a selection means for selecting a first CG model having a shape substantially identical to that of the first object based on the three-dimensional positions of the plurality of markers; an estimation means for estimating a relative position and orientation relationship between the first CG model selected by the selection means and the first object based on three-dimensional positions of the plurality of markers; a placement means for placing the first CG model so as to overlap the first object in the specific space based on the position and orientation relationship estimated by the estimation means; A placement device comprising:

2. the selection means selects the first CG model from among a plurality of CG models based on the three-dimensional positions of the plurality of markers.

2. The placement device according to claim 1 .

3. the selection means selects the first CG model based on a volume of the first object and volumes of the plurality of CG models.

3. The arrangement device according to claim 2.

4. 4. The arrangement device according to claim 3, wherein the selection means 1) calculates the volume of the smallest rectangular parallelepiped containing the plurality of markers based on the three-dimensional positions of the plurality of markers as the volume of the first object, and 2) calculates the volume of the smallest rectangular parallelepiped containing each vertex of the CG model as the volume of the CG model.

5. the selection means selects the first CG model based on a ratio of sides of a smallest rectangular parallelepiped that contains the plurality of markers and a ratio of sides of a smallest rectangular parallelepiped that contains each vertex of the CG model.

5. An arrangement device according to claim 2, wherein the arrangement device is a casing.

6. the estimation means estimates a relative position and orientation relationship between the first CG model and the first object such that a sum of distances between a polygonal surface of the first CG model and each of a plurality of markers arranged on the first object is minimum; 5. An arrangement device according to claim 1 , wherein the arrangement device is a casing.

7. The position acquisition means acquires three-dimensional positions of the plurality of markers based on an image of the first object captured by an imaging device.

5. An arrangement device according to claim 1 , wherein the arrangement device is a casing.

8. 8. The placement device according to claim 7, further comprising a determination means for, when the first object moves, determining all markers that move with the first object as the plurality of markers placed on the first object.

9. the position acquisition means acquires three-dimensional positions of the plurality of markers and three-dimensional positions of one or a plurality of feature points on the first object based on an image of the first object captured by an imaging device; the estimation means estimates a relative position and orientation relationship between the first CG model selected by the selection means and the first object, based on the three-dimensional positions of the plurality of markers and the three-dimensional positions of the one or more feature points; 5. An arrangement device according to claim 1 , wherein the arrangement device is a casing.

10. the position acquisition means acquires three-dimensional positions of the plurality of markers and three-dimensional positions of one or a plurality of feature points on the first object based on a plurality of images captured by the imaging device of the first object; the estimation means estimates a relative position and orientation relationship between the first CG model selected by the selection means and the first object, based on a three-dimensional position of a feature point that has been detected a predetermined number of times among the one or more feature points and the three-dimensional positions of the plurality of markers; 10. The placement device of claim 9.

11. A method for arranging a CG model having a shape substantially identical to that of an object in a specific space so as to overlap the object, comprising the steps of: a position acquisition step of acquiring three-dimensional positions of a plurality of markers arranged on a first object; a selection step of selecting a first CG model having a shape substantially identical to that of the first object based on the three-dimensional positions of the plurality of markers; an estimation step of estimating a relative position and orientation relationship between the first CG model selected in the selection step and the first object based on three-dimensional positions of the plurality of markers; a placement step of placing the first CG model in the specific space so as to overlap with the first object, based on the position and orientation relationship estimated in the estimation step; 13. An arrangement method comprising:

12. A program for causing a computer to function as each of the means of the arrangement device according to any one of claims 1 to 4.

Images