Mixed reality system, program, mobile terminal device and method
By arranging a set of feature points in the real space and identifying their locations, the problem that the prior art is difficult to realize mixed reality in a wide space is solved, and the image display of virtual objects and real environments is realized, which improves the visibility of virtual objects and the visibility of virtual objects and Interactive.
Patent Information
- Application Number
- CN201980069925.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-24
- Filing Date
- 2019-08-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-08-23
AI Technical Summary
Existing mixed reality technologies are difficult to realize the real experience of virtual objects in broad spaces, such as virtual giant robots cannot fly between buildings in outdoor spaces.
By arranging a plurality of feature points sets in the real space and identifying the position and posture information of these feature points sets using the mobile terminal device, the viewpoint position of the virtual camera is determined, thereby displaying a fused image of the virtual object and the real space on the display.
It realizes a mixed reality experience in a wide space such as outdoor space, allowing users to see a composite image of virtual objects and real environments, and improves the visibility and interactivity of virtual objects in real space.
Smart Images

Figure CN112912936B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a mixed reality system, etc., and in particular to a mixed reality system, etc. that enables a user existing in a prescribed real space to experience mixed reality. Background Art
[0002] Recently, mixed reality (MR) technology for seamlessly merging the real world with the virtual world in real time has been known. MR technology enables a user who is experiencing MR technology to have an experience as if a virtual object exists in the place. Regarding MR technology, a technology for performing MR itself with high accuracy so that objects in a virtual space are displayed without causing discomfort to the user has been developed. For example, Patent Document 1 discloses a system capable of recognizing a real space environment in a mixed reality environment in real time.
[0003] {Citation List}
[0004] {Patent Documents}
[0005] {Patent Document 1}
[0006] Japan Special Opening 2017-227975
[0007] {Non-patent document 1}
[0008] Francisco J. Romero-Ramirez, Rafael Munoz-Salinas, Rafael Medina-Carnicer, "Speeded up detection of squared fiducial markers," Image and Vision Computing, Volume 76, 2018, Pages 38-47, ISSN 0262-8856, DOI: 10.1016 / j.imavis.2018.05.004.
[0009] {Non-patent document 2}
[0010] David G Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, Vol. 50, No. 2, 2004, pp. 91-110.
[0011] {Non-Patent Document 3}
[0012] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski, “ORB: and efficient alternative to SIFT or SURF,” IEEE International Conference on Computer Vision, 2011.
[0013] {Non-Patent Document 4}
[0014] Genshiro Kitagawa, "Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models", Journal of Computational and Graphical Statistics, Vol. 5, No. 1 (March 1996), pp. 1-25.
[0015] {Non-Patent Document 5}
[0016] Shuichi Kurabayashi, Hidetoshi Ishiyama, and Masaru Kanai, "Sensing By Overlaying: a Practical Implementation of a Multiplayer Mixed-Reality Gaming System by Integrating a Dense Point Cloud and a Real-Time Camera," in Proceedings of the 2016 IEEE International Symposium on Multimedia (ISM), pp. 636-639, San Jose, CA, DOI: 10.1109 / ISM.2016.0136, (2016). Summary of the invention
[0017] Problem that the invention aims to solve
[0018] Conventionally, the target of MR technology is the real space within the sensor observation range, and it is difficult to realize MR in a wide space. For example, in an outdoor space, it is difficult to realize MR in which a virtual giant robot flies between buildings.
[0019] The present invention is made to solve the above-mentioned problems, and its main purpose is to provide a mixed reality system that enables users to experience MR corresponding to a place in a relatively wide space such as an outdoor space.
[0020] Solutions for solving problems
[0021] In order to achieve the above-mentioned purpose, a system according to one aspect of the present invention is a mixed reality system for displaying a mixed reality image on a display, wherein the display is used to display virtual objects to a user existing in a specified real space, wherein in the mixed reality image, images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, and the mixed reality system includes a mobile terminal device having the display and a shooting device for shooting the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; and the mobile terminal device performs the following operations: stores pre-acquired data of a first virtual object and data of a second virtual object in the virtual space that does not correspond to the real object, and stores data for being arranged in The method comprises the following steps: obtaining, in a virtual space, layout position and posture information of each feature point set in the specified real space, the first virtual object corresponding to the real object existing in the specified real space and defining the virtual space, identifying each feature point set photographed by the photographing device, and determining the viewpoint position of the virtual camera in the virtual space corresponding to the position and shooting direction of the photographing device in the real space based on the layout position and posture information for each feature point set and the relative position and posture information of the mobile terminal device relative to each feature point set, the layout position and posture information being obtained from identifiable information of a part or all of the feature point sets in the identified feature point sets, the relative position and posture information being determined from the shape and size of the feature point sets, and generating, based on the data of the first virtual object, the data of the second virtual object and the viewpoint position, a mixed reality image in which the image of the second virtual object corresponding to the viewpoint position is superimposed on the photographed image in the real space.
[0022] In addition, in the present invention, preferably, when the first virtual object and the second virtual object overlap with each other in the field of view of the virtual camera when generating the mixed reality image, if the depth distance of the second virtual object relative to the viewpoint position is greater than the depth distance of the first virtual object relative to the viewpoint position, the mobile terminal device generates a mixed reality image superimposed with an image of the second virtual object with the overlapping portion removed.
[0023] In addition, in the present invention, preferably, the mixed reality system includes five or more feature point sets, and the mobile terminal device identifies each feature point set photographed by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image.
[0024] In addition, in the present invention, preferably, the mixed reality system includes four feature point sets arranged in the specified real space, each feature point set has a positional relationship corresponding to the four vertex positions of the virtual rectangle in the real space, and each feature point set is arranged to be away from each other by a predefined distance in the vertical direction and away from each other by a predefined distance in the horizontal direction.
[0025] Furthermore, in the present invention, preferably, one or more of the feature point sets are fixedly attached to each of a plurality of rod-shaped bodies installed in a manner isolated in a horizontal direction.
[0026] Furthermore, in the present invention, preferably, each characteristic point set is fixedly attached to a transmissive member formed of a light-transmitting material.
[0027] Furthermore, in the present invention, preferably, each feature point set is fixedly attached by using a line.
[0028] Furthermore, in the present invention, preferably, the feature point set is a poster, a painting, a picture, an ArUco marker, an AR marker, a QR code or a data matrix.
[0029] In order to achieve the above-mentioned purpose, a program according to one aspect of the present invention is a program executed by a mobile terminal device in a mixed reality system, wherein the mixed reality system is used to display a mixed reality image on a display, and the display is used to display virtual objects to users existing in a specified real space, in which the images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on the captured image of the real space, and the mixed reality system includes the mobile terminal device having the display and a shooting device for shooting the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; and the program enables the mobile terminal device to perform the following steps: identify the shooting device The method comprises the following steps: placing each captured feature point set, determining the viewpoint position of the virtual camera in the virtual space corresponding to the position and shooting direction of the shooting device in the real space based on the arrangement position and posture information in the virtual space where the feature point sets are arranged, and the relative position and posture information of the mobile terminal device relative to the feature point sets, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the identified feature point sets, the relative position and posture information is determined from the shape and size of the feature point sets, and based on the pre-acquired data of a first virtual object, the data of a second virtual object in the virtual space that does not correspond to the real object, and the viewpoint position, generating a mixed reality image in which the image of the second virtual object corresponding to the viewpoint position is superimposed on the captured image in the real space, wherein the first virtual object corresponds to the real object existing in the specified real space and defines the virtual space.
[0030] In addition, in order to achieve the above-mentioned purpose, a mobile terminal device according to one aspect of the present invention is a mobile terminal device in a mixed reality system, wherein the mixed reality system is used to display a mixed reality image on a display, and the display is used to display virtual objects to users existing in a specified real space, and in the mixed reality image, the image of the virtual object arranged in the virtual space corresponding to the real space is superimposed on the captured image of the real space, and the mixed reality system includes the mobile terminal device having the display and a shooting device for shooting the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and the mobile terminal device performs the following operations: stores pre-acquired data of a first virtual object, and data of a second virtual object in the virtual space that does not correspond to a real object data, and stores arrangement position and posture information in a virtual space for arranging each feature point set in the specified real space, the first virtual object corresponds to a real object existing in the specified real space and defines a virtual space, identifies each feature point set photographed by the photographing device, and determines the viewpoint position of the virtual camera in the virtual space corresponding to the position and shooting direction of the photographing device in the real space based on the arrangement position and posture information for each feature point set and the relative position and posture information of the mobile terminal device relative to each feature point set, the arrangement position and posture information is obtained from identifiable information of a part or all of the identified feature point sets, the relative position and posture information is determined from the shape and size of the feature point sets, and based on the data of the first virtual object, the data of the second virtual object and the viewpoint position, generates a mixed reality image in which the image of the second virtual object corresponding to the viewpoint position is superimposed on the photographed image in the real space.
[0031] In addition, in order to achieve the above-mentioned purpose, a method according to one aspect of the present invention is a method performed by a mobile terminal device in a mixed reality system, wherein the mixed reality system is used to display a mixed reality image on a display, and the display is used to display virtual objects to users existing in a specified real space, and in the mixed reality image, images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on the captured image of the real space, and the mixed reality system includes the mobile terminal device having the display and a shooting device for shooting the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; and the method includes the following steps: identifying the image captured by the shooting device The viewpoint position of the virtual camera in the virtual space corresponding to the position and shooting direction of the shooting device in the real space is determined based on the arrangement position and posture information in the virtual space where the feature point sets are arranged, and the relative position and posture information of the mobile terminal device relative to the feature point sets, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the identified feature point sets, the relative position and posture information is determined from the shape and size of the feature point sets, and based on the pre-acquired data of a first virtual object, the data of a second virtual object in the virtual space that does not correspond to the real object, and the viewpoint position, a mixed reality image is generated in which an image of a second virtual object corresponding to the viewpoint position is superimposed on the captured image in the real space, wherein the first virtual object corresponds to a real object existing in the specified real space and defines the virtual space.
[0032] In addition, in order to achieve the above-mentioned purpose, a system according to one aspect of the present invention is a mixed reality system for displaying a mixed reality image on a display, wherein the display is used to display virtual objects to a user existing in a specified real space, in which the image of the virtual object arranged in the virtual space corresponding to the real space is superimposed on the captured image of the real space, and the mixed reality system includes a server and a mobile terminal device, wherein the mobile terminal device has the display and a shooting device for shooting the real space, and is characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; the server stores pre-acquired data of a first virtual object and data of a second virtual object in the virtual space that does not correspond to the real object, and stores data for being arranged in the specified real space. The mobile terminal device performs the following operations: identifying each feature point set photographed by the photographing device, and determining the viewpoint position of the virtual camera in the virtual space corresponding to the position and shooting direction of the photographing device in the real space based on the layout position posture information for each feature point set and the relative position posture information of the mobile terminal device relative to each feature point set, the layout position posture information being obtained from identifiable information of a part or all of the feature point sets in the identified feature point set, the relative position posture information being determined from the shape and size of the feature point set, and generating a mixed reality image based on the data of the first virtual object, the data of the second virtual object and the viewpoint position, in which the image of the second virtual object corresponding to the viewpoint position is superimposed on the photographed image in the real space.
[0033] In addition, in order to achieve the above-mentioned purpose, a system according to one aspect of the present invention is a mixed reality system for displaying a mixed reality image on a display, wherein the display is used to display virtual objects to a user existing in a specified real space, in which the image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on the captured image of the real space, and the mixed reality system includes a mobile terminal device having the display and a shooting device for shooting the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; and the mobile terminal device performs the following operations: stores pre-acquired data of a first virtual object, and data in the virtual space that does not correspond to a real object The invention relates to a method for manufacturing a mixed reality image of a mobile terminal device and a second virtual object, wherein the method comprises: obtaining a mixed reality image of the mobile terminal device and a second virtual object, and storing layout position and posture information in the virtual space for arranging each feature point set in the specified real space, wherein the first virtual object corresponds to a real object existing in the specified real space and defines a virtual space, identifying each feature point set photographed by the photographing device, and determining the position and posture information of the first virtual object based on the layout position and posture information for each feature point set and the relative position and posture information of the mobile terminal device relative to each feature point set, wherein the layout position and posture information is obtained from identifiable information of a part or all of the feature point sets in the identified feature point sets, and the relative position and posture information is determined from the shape and size of the feature point sets, and generating a mixed reality image in which the image of the second virtual object corresponding to the viewpoint position is superimposed on the photographed image in the real space based on the data of the first virtual object and the data of the second virtual object.
[0034] In addition, in order to achieve the above-mentioned purpose, a program according to one aspect of the present invention is a program executed by a mobile terminal device in a mixed reality system, wherein the mixed reality system is used to display a mixed reality image on a display, and the display is used to display virtual objects to a user existing in a specified real space, and in the mixed reality image, the image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on the captured image of the real space, and the mixed reality system includes the mobile terminal device having the display and the shooting device for shooting the real space, and is characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the specified real space, the feature point sets include identifiable information that enables each feature point set to be identified, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; and the program enables the mobile terminal device to The mobile terminal device performs the following steps: identifying each feature point set photographed by the photographing device, determining the position and posture information of the first virtual object obtained in advance based on the layout position and posture information in the virtual space where each feature point set is arranged, and the relative position and posture information of the mobile terminal device relative to each feature point set, wherein the layout position and posture information is obtained from identifiable information of a part or all of the feature point sets in the identified feature point set, the relative position and posture information is determined from the shape and size of the feature point set, the first virtual object corresponds to a real object existing in the specified real space and defines the virtual space, and based on the data of the first virtual object and the data of the second virtual object in the virtual space that does not correspond to the real object, a mixed reality image is generated in which the image of the second virtual object corresponding to the viewpoint position is superimposed on the captured image in the real space.
[0035] Effects of the Invention
[0036] According to the present invention, a user can experience MR corresponding to a place in a relatively wide space such as an outdoor space. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is a schematic diagram showing a portion of a mixed reality system according to an embodiment of the present invention.
[0038] Figure 2 is a block diagram showing a hardware configuration of a mobile terminal device according to an embodiment of the present invention.
[0039] Figure 3 is a diagram showing a two-dimensional code according to an embodiment of the present invention.
[0040] Figure 4 is a schematic diagram showing a portion of a real space according to an embodiment of the present invention.
[0041] Figure 5 is shown with Figure 4 A diagram of the virtual space corresponding to the real space shown in .
[0042] Figure 6 is a functional block diagram of a mobile terminal device according to an embodiment of the present invention.
[0043] Figure 7 is an example of an image of a second virtual object.
[0044] Figure 8 yes Figure 7 An example of a mixed reality image in which an image of a second virtual object is superimposed on a captured image of a real space is shown in FIG.
[0045] Fig. 9 It is a diagram for explaining the positional relationship between a mobile terminal device and a feature point set included in a feature point set group in a mixed reality system according to an embodiment of the present invention.
[0046] Fig.10 is a flow chart illustrating information processing in a mixed reality system according to an embodiment of the present invention.
[0047] Fig.11 is a diagram showing a feature point cluster 3 fixedly attached to a window glass.
[0048] Fig.12 An example of the overall configuration of a mixed reality system according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0049] Now, a mixed reality (MR) system that provides a user with a mixed reality space in which a virtual space and a real space are mixed together will be described with reference to the accompanying drawings. In this specification, for the sake of convenience of description, unnecessary more detailed descriptions may be omitted. For example, detailed descriptions related to well-known matters or repeated descriptions related to substantially the same configuration may be omitted. Note that unless otherwise specifically stated, in each figure, the same symbol indicates the same or corresponding part.
[0050] The MR system according to the present embodiment allows a user to experience MR by drawing virtual objects, which are objects arranged in a virtual space corresponding to the real space, in a superimposed manner on a captured real space image that the user visually recognizes through, for example, a screen of a smartphone.
[0051] Figure 1is a schematic diagram showing a portion of a mixed reality system (MR system) 1 according to an embodiment of the present invention. The MR system 1 according to an embodiment of the present invention is implemented in a predefined prescribed real space (real space 50). The real space 50 is a predefined outdoor real space, and real objects that are objects in the real world exist in the space. For example, real objects are structures such as buildings, bridges, and walls that are fixed in the real space. However, moving items may be included in the real objects. In addition, instead of the outdoor real space, the real space 50 may be an indoor real space, and may partially include the indoor real space.
[0052] like Figure 1 As shown in , the MR system 1 includes a mobile terminal device 2 and a feature point set group 3 including a plurality of feature point sets 4 arranged in a real space 50 .
[0053] Figure 2 1 is a block diagram showing the hardware configuration of a mobile terminal device 2 according to an embodiment of the present invention. The mobile terminal device 2 includes a processor 11, an input device 12, a display device 13, a camera 14, a storage device 15, and a communication device 16. These components are connected via a bus 17. Note that an interface is inserted between the bus 17 and each component as needed. In the present embodiment, the mobile terminal device 2 is a smart phone. However, the mobile terminal device 2 may be a terminal such as a tablet computer or a computer equipped with a contact-type input device such as a touch pad, as long as the terminal is a portable terminal having the above configuration.
[0054] The processor 11 controls the overall operation of the mobile terminal device 2. For example, the processor 11 is a CPU. Alternatively, an electronic circuit such as an MPU or the like may be used as the processor 11. The processor 11 performs various processes by loading a program and data stored in the storage device 15 and executing the program. In one example, the processor 11 is composed of a plurality of processors.
[0055] The input device 12 is a user interface for accepting input from the user to the mobile terminal device 2. The display device (display) 13 displays an application screen or an image captured by the camera 14 to the user of the mobile terminal device 2 according to the control of the processor 11. In the present embodiment, the mobile terminal device 2 as a smart phone includes a touch screen as the input device 12, the touch screen is also used as the display device 13, and the input device 12 and the display device 13 are constructed in an integrated form. However, the input device 12 and the display device 13 may be arranged at different positions as separate units. In this case, the input device 12 may be, for example, a touch pad or a button, and the display device 13 may be a liquid crystal display, a display using an organic EL, a plasma display, or the like.
[0056] The photographing device (camera) 14 photographs (captures) a still image or a moving image of a real space, and stores the photographed image or moving image data in the storage device 15. The photographing device 14 is, for example, a camera constituted by an image sensor.
[0057] The storage device 15 is a storage device provided in a general smartphone, and includes a RAM as a volatile memory and a ROM as a nonvolatile memory. The storage device 15 may include an external memory.
[0058] In one example, the storage device 15 includes a main storage device and an auxiliary storage device. The main storage device is a volatile storage medium that allows high-speed reading and writing of information, and is used as a storage area and a work area when the processor 11 processes information. The auxiliary storage device stores various programs and data used by the processor 11 when executing various programs. The auxiliary storage device is, for example, a hard disk device, but can be any kind of non-volatile storage device or non-volatile memory capable of storing information, and can be of a removable type. For example, the auxiliary storage device stores an operating system (OS), middleware, application programs, various data that can be referenced when executing these programs, etc.
[0059] The communication device 16 sends data to other computers and receives data from other computers via a network such as the Internet. For example, the communication device 16 performs wireless communication such as mobile communication or wireless LAN communication to connect to the network. In one example, the communication device 16 downloads a program from a server and stores the program in the storage device 15. However, the communication device 16 can perform wired communication using an Ethernet (registered trademark) cable or the like. In the case where data is not sent to or received from other computers, the mobile terminal device 2 does not need to include the communication device 16.
[0060] The feature point set 4 represents a local feature point set, and in the present embodiment, the feature point set 4 is an ArUco tag (non-patent document 1). The feature point set 4 is pre-arranged and fixed in the real space 50. Each feature point set 4 is associated with unique identification information (ID). However, it is sufficient that each feature point set 4 includes identifiable information that enables identification of each feature point set 4. The identification information is an example of identifiable information. In addition, the feature point set 4 is not limited to the ArUco tag.
[0061] Figure 3is a diagram showing a feature point set 4 according to an embodiment of the present invention. The feature point set 4 is composed of white squares and black squares, and has a square shape as a whole. The black squares are arranged at positions for constituting the outer edges of the feature point set 4. The white squares and the black squares are arranged at different positions in each feature point set 4, and an ID (e.g., an ID number) is assigned according to the arrangement position. For example, the feature point set 4 is made by printing on a metal plate with a flat surface using a pigment, or attaching the printed metal plate to other supports, and the metal plate is coated to prevent sunlight reflection.
[0062] It is sufficient that the feature point set 4 enables each feature point set 4 to be identified and configured and fixed in the real space 50, and therefore is not limited to the above example. The feature point set 4 can be made by printing the pattern of the feature point set 4 itself, or can be made by including the pattern of the feature point set 4 itself in the print. For example, the feature point set 4 has a square shape with a side length of 10 cm.
[0063] like Figure 1 As shown in , the feature point set group 3 includes four feature point sets 4, and each feature point set 4 is fixedly attached to each of a pair of rod-shaped bodies 41 installed in a manner isolated in the horizontal direction in a manner isolated in the vertical direction. For example, the rod-shaped body 41 is a fixed rod or column, and is fixedly attached to the ground. Each feature point set 4 included in the feature point set group 3 has a positional relationship corresponding to the four vertex positions of the virtual rectangle in the real space 50, and is arranged to be away from each other by a predefined distance in the vertical direction and away from each other by a predefined distance in the horizontal direction. Preferably, the two sides constituting the virtual rectangle are parallel to each other in the vertical direction (vertical direction) and the horizontal direction. Note that the feature point set group 3 indicates a plurality of feature point sets 4. The feature point set group 3 may indicate a plurality of feature point sets 4 that exist locally, or may indicate a plurality of feature point sets 4 included in a single captured image captured by the capturing device 14. In addition, the MR system 1 may include a plurality of feature point set groups 3.
[0064] In the present embodiment, the real space 50 is an area having a square range of several kilometers, such as the entire area around Shibuya Station. Figure 4 1 is a schematic diagram showing a part of a real space 50 according to an embodiment of the present invention. The real space 50 includes a real object 51 such as a building.
[0065] The virtual space 60 is a virtual space associated with the real space 50, and the position in the real space 50 and the position in the virtual space 60 are associated with each other in a one-to-one manner. In the virtual space 60, the MR system 1 uses three-dimensional space data for representing the real object 51 existing in the real space 50. Therefore, the three-dimensional space data defines the virtual space 60 associated with the real space 50 and represents the three-dimensional shape of the real object 51. The three-dimensional space data is obtained in advance, and each item of the three-dimensional space data is composed of a three-dimensional shape element, which is a basic unit having three-dimensional position information in the virtual space 60. This basic unit will be referred to as a three-dimensional shape element. In the present embodiment, a three-dimensional mesh is used as a three-dimensional shape element. Therefore, the three-dimensional space data is composed of a three-dimensional mesh. The three-dimensional mesh is formed by one or more polygons, and at least one of the vertices of the polygons constituting each three-dimensional mesh has three-dimensional position information. However, the three-dimensional shape element may be point group data or the like.
[0066] The three-dimensional position information possessed by the three-dimensional shape element is the position information in the real space 50 corresponding to the position information in the virtual space 60. Here, the position information in the real space 50 is, for example, a 6DoF (6-axis) value including latitude, longitude, altitude, and direction, and the position information in the virtual space 60 is also a corresponding 6DoF value.
[0067] Recently, a number of different companies provide three-dimensional mesh data of a relatively wide area (which is the area of interest for the real space 50 in the present embodiment), and accurate three-dimensional mesh data can be obtained relatively easily.
[0068] For example, the three-dimensional mesh data as described above is obtained in advance as follows: First, point group data indicating the three-dimensional shape of the real object 51 in the real space 50 is obtained in advance using a high-precision laser scanner. Figure 5 An example of point group data in a virtual space 60 associated with a real space 50 is shown. For example, each item of the point group data has a three-dimensional coordinate (x, y, z) and is arranged in a virtual space 60 associated with the real space 50. Note that each item of the point group data is color point group data including color information. Next, based on the color point group data obtained as described above, polygons having each point group as a vertex are formed by using a known method. A three-dimensional mesh is composed of polygons. However, since the three-dimensional shape elements constituting the three-dimensional space data do not need to have color information, in the above example, the point group data does not need to have color information. In other examples, a three-dimensional mesh is generated based on multiple captured images in the real space 50 by using a known method.
[0069] Figure 5 Shown with Figure 4, and also shows a first virtual object 61, which is a virtual object corresponding to the real object 51 existing in the real space 50. Although all objects arranged in the virtual space 60 are virtual objects, for the sake of convenience of description, the virtual object corresponding to the real object 51 will be referred to as the first virtual object 61, and the virtual object that does not correspond to the real object 51 (that is, the virtual object that does not exist in the real space 50) will be referred to as the second virtual object 62. Therefore, the three-dimensional space data is the data of the first virtual object 61, and the virtual space 60 is defined by the data of the first virtual object 61. In the present embodiment, for example, the coordinates of the three-dimensional space data have the same meaning as the coordinates of the virtual space 60.
[0070] like Figure 5 As shown in FIG. 6 , a position in a virtual space 60 is determined by using 6DoF including three-dimensional coordinates (x, y, z). However, due to Figure 5 The three-dimensional coordinates in the virtual space 60 shown in are examples, so the origin of the three-dimensional coordinates can be set to any position, and the respective coordinate axes can have any direction as long as the coordinate axes are orthogonal to each other. Alternatively, with respect to the position in the virtual space 60, the three-dimensional position can be determined by using other types of coordinate systems.
[0071] Figure 6 1 is a functional block diagram of a mobile terminal device 2 according to an embodiment of the present invention. The mobile terminal device 2 includes an application unit 21. The application unit 21 includes an identification unit 22, a position determination unit 23, an image generation unit 24, and an image display unit 25. In the present embodiment, these functions are implemented by executing a program by the processor 11. In the present embodiment, since various functions are implemented by loading a program, a part of a component (function) may be included in other components. However, these functions may be implemented by hardware by configuring an electronic circuit or the like for implementing a part or all of each function.
[0072] An OS is installed on the mobile terminal device 2, and an application that runs on the OS for allowing a user to experience MR is also installed. The application unit 21 corresponds to the application, and is stored in the storage device 15 as an application program.
[0073] The storage device 15 stores data of a first virtual object 61 in a virtual space 60 corresponding to a real object 51 existing in the real space 50, and data of a second virtual object 62 in the virtual space 60. In the present embodiment, the data of the second virtual object 62 is also formed by a three-dimensional grid, and the position information included in the data of the second virtual object is represented by the coordinates of the three-dimensional space data or is associated with the coordinates of the three-dimensional space data. In one example, the second virtual object 62 is a virtual character or a virtual item. Figure 7 is an example of an image of the second virtual object 62. In one example, the data of the second virtual object 62 is data of a virtual object whose at least one of shape, size, or orientation changes according to at least one of position and time.
[0074] The storage device 15 stores feature point set position data in which the ID of each feature point set 4 is associated with the layout position information in the virtual space 60, which corresponds to the position information in the real space 50 where each feature point set 4 is arranged. The layout position information here is represented by the coordinates of the three-dimensional space data or is associated with the coordinates of the three-dimensional space data. In addition, more accurately, the layout position information is the layout position posture information, and is, for example, a 6DoF (six degrees of freedom) value. Therefore, for example, the layout position information (layout position posture information) is the position posture information of the feature point set 4 using the prescribed position of the first virtual object 61 (the origin of the virtual space 60) as the origin. Note that, similar to the case where ArUco is usually used, the storage device 15 stores a software module for using ArUco, and the application unit 21 uses the module to associate each feature point set 4 with an ID.
[0075] In one example, the storage device 15 stores the three-dimensional space data together with a program for using the three-dimensional space data as a three-dimensional space database. For example, in this case, the application unit 21 uses the combination of IDs of the identified feature point sets 4 to query the three-dimensional space database, thereby obtaining the three-dimensional space data based on the combination of the IDs.
[0076] The application unit 21 starts the photographing device 14 when the application is started by a user operation or the like. The recognition unit 22 recognizes each feature point set 4 photographed by the photographing device 14. For example, the recognition unit 22 recognizes the feature point set 4 based on the outer edge of the feature point set 4 and the arrangement positions of the white squares and the black squares of the feature point set 4, and recognizes the ID of the feature point set 4. Preferably, the recognition unit 22 recognizes each feature point set 4 photographed by the photographing device 14 at a prescribed time interval such as 0.5 s.
[0077] The position determination unit 23 determines a virtual camera viewpoint position in the virtual space 60, which corresponds to the position and the shooting direction of the camera 14 in the real space 50. For example, the virtual camera viewpoint position determined by the position determination unit 23 is represented by 6DoF (six degrees of freedom) in the virtual space 60. For example, the virtual camera angle of view is determined according to the actual angle of view of the camera 14.
[0078] In one example, the application unit 21 determines whether the number of feature point sets 4 captured by the capturing device 14 is greater than or equal to a prescribed number, and in the case where it is determined that the number of feature point sets 4 captured by the capturing device 14 is greater than or equal to the prescribed number, the position determination unit 23 determines the virtual camera viewpoint position. For example, in the case where the application unit 21 determines that the recognition unit 22 has recognized three or more feature point sets 4, the position determination unit 23 determines the virtual camera viewpoint position.
[0079] The position determination unit 23 obtains arrangement position information related to the arrangement of the feature point set 4 by using the ID of the identified feature point set 4 and the feature point set position data stored in the storage device 15. In addition, the position determination unit 23 determines relative position information indicating the position of the mobile terminal device 2 relative to the identified feature point set 4 according to the shape and size of the feature point set 4. More precisely, the relative position information is relative position posture information (an example of the relative position information is information including 6DoF as the relative position), and the relative position information includes the three-dimensional position and facing direction of the mobile terminal device 2 (photographing device 14) relative to the feature point set 4. Therefore, for example, the relative position information (relative position posture information) is the position posture information of the feature point set 4 using the virtual camera position as the origin. The position determination unit 23 obtains the arrangement position information for each of the identified feature point sets 4, and determines the relative position information indicating the position of the mobile terminal device 2 relative to the feature point set 4. The position determination unit 23 determines the virtual camera viewpoint position (6DoF) based on the obtained arrangement position information and the determined relative position information.
[0080] In one example, the position determination unit 23 calculates a tentative virtual camera viewpoint position (6DoF) by using the arrangement position information and relative position information of each of the identified feature point sets 4. The position determination unit 23 determines the virtual camera viewpoint position by using the average value of the calculated multiple 6DoF values. Alternatively, the position determination unit 23 determines the virtual camera viewpoint position by using the median value of the calculated multiple 6DoF values. Alternatively, the position determination unit 23 determines the virtual camera viewpoint position by inputting the calculated multiple 6DoF values into an estimation model such as a particle filter (Non-Patent Document 4).
[0081] In one example, instead of the above example, or in addition to the above example, the position determination unit 23 determines the virtual camera viewpoint position by using the arrangement relationship between the known feature point sets 4. Here, since the feature point sets 4 are arranged away from each other by a predefined distance in the vertical direction and the horizontal direction, the position of the other feature point sets 4 can be derived from the position of one feature point set 4. In one example, the position determination unit 23 determines relative position information for each identified feature point set 4, and then determines a first virtual rectangle composed of the identified feature point sets 4. On the other hand, the position determination unit 23 obtains arrangement position information for each identified feature point set 4, and then determines a second virtual rectangle composed of the obtained arrangement position information. The position determination unit 23 determines the virtual camera viewpoint position by using the arrangement position information of the second virtual rectangle and the shape and size of the first virtual rectangle relative to the second virtual rectangle.
[0082] In other examples, in the case where the tentative virtual camera viewpoint position is set to a 6DoF value calculated by using the arrangement position information and relative position information of one feature point set 4, the position determination unit 23 calculates the difference between the position of the other feature point set 4 derived from the one feature point set 4 and the position of the other feature point set 4 recognized from the image captured by the camera 14. The position determination unit 23 extracts the 6DoF value whose calculated difference is within the prescribed range from the calculated respective 6DoF values, and determines the virtual camera viewpoint position by using the average value or median value of the 6DoF values or by inputting the 6DoF values into a known estimation model. Alternatively, the position determination unit 23 weights the calculated respective 6DoF values according to the calculated difference, and determines the virtual camera viewpoint position by using the average value or median value of the weighted 6DoF values or by inputting the 6DoF values into a known estimation model. Alternatively, the position determination unit 23 determines the virtual camera viewpoint position based on the calculated 6DoF value, and then corrects the virtual camera viewpoint position by using the arrangement relationship of the known feature point set group 3.
[0083] In one example, the position determination unit 23 determines the position of the mobile terminal device 2 relative to the feature point set 4 as follows. The size and shape of each feature point set 4 are predefined. By using a known method, the position determination unit 23 recognizes the feature point set 4 as an image, recognizes the outline of the feature point set 4, and calculates a coordinate transformation matrix for converting from the feature point set coordinate system to the camera coordinate system. By using the calculated coordinate transformation matrix, the position determination unit 23 calculates the position and posture of the feature point set 4 in the image captured by the camera 14 in the camera coordinate system, and determines the position of the feature point set 4 relative to the camera 14 (6DoF).
[0084] In one example, the position determination unit 23 determines the position of the mobile terminal device 2 relative to the feature point set 4 as follows. The position determination unit 23 uses a known method (non-patent document 2 and non-patent document 3) in which local features that are robust features against rotation and movement are used to detect objects included in the captured image. In this example, the storage device 15 stores the template images of each feature point set 4 in association with the ID of the feature point set 4. The position determination unit 23 determines the relative position (6DoF) by using local features to derive projections (such as rotation, enlargement / reduction, and deformation) between the template image and the image captured by the camera 14. Here, local features refer to patterns or significant structures seen in the image, such as points, edges, or small image blocks (patches). Specifically, the projection is estimated by using the points at the four corners of the template image and the homography matrix obtained by using local features. The homography matrix refers to the parameters used when a plane is projected onto another plane by a projection transformation.
[0085] Although the virtual camera viewpoint position is calculated and determined by using 6DoF, other formats of position information may be used as long as equivalent information is included.
[0086] Based on the data of the first virtual object 61 and the data of the second virtual object 62 stored in the storage device 15 and the virtual camera viewpoint position determined by the position determination unit 23, the image generation unit 24 generates a mixed reality image in which the image of the second virtual object 62 according to the viewpoint position is superimposed on the captured image of the real space 50. The image display unit 25 displays the application screen including the generated mixed reality image on the display device 13.
[0087] In one example, the image generation unit 24 reads data of the first virtual object 61 and data of the second virtual object 62 corresponding to the virtual camera viewpoint position determined by the position determination unit 23, and generates a mixed reality image in which the image of the second virtual object 62 according to the viewpoint position is superimposed on the captured image of the real space 50. When generating the mixed reality image, a known method as indicated in Patent Document 1 or Non-Patent Document 5 can be used.
[0088] Figure 8 yes Figure 7 An example of a mixed reality image in which an image of the second virtual object 62 is superimposed on a captured image of the real space 50 is shown. In a case where the first virtual object 61 and the second virtual object 62 overlap each other in the field of view range of the virtual camera, if the depth distance of the second virtual object 62 relative to the virtual camera viewpoint position is greater than the depth distance of the first virtual object 61 relative to the virtual camera viewpoint position, the image generation unit 24 generates a mixed reality image in which the image of the second virtual object 62 except for the overlapping portion is superimposed. For example, the image generation unit 24 performs a drawing process of the first virtual object 61 as an invisible mesh, and performs a normal drawing process for the second virtual object 62. By doing so, for the above-mentioned overlapping portion, an invisible drawing process is performed for the first virtual object 61 that is closer to the virtual camera viewpoint position than the second virtual object 62. Note that the field of view range of the virtual camera is determined according to the viewing angle of the camera 14.
[0089] Figure 8 A second virtual object 62 is shown, and a portion of the second virtual object 62 is not shown. Figure 8 , it can be seen that the second virtual object 62 is displayed such that the second virtual object 62 exists behind the building which is the real object 51.
[0090] In one example, the image generation unit 24 generates an MR motion image by generating a mixed reality image at a specified frame rate. In one example, when the shooting device 14 does not recognize the feature point set 4 or when the virtual camera viewpoint position is not determined, the image display unit 25 directly displays the shot image of the real space 50 as a mixed reality image.
[0091] In the present embodiment, due to the operation of various parts of the application unit 21, the MR system 1 needs to include at least three feature point sets 4 in a single captured image captured by the capturing device 14. Based on the viewing angle of the capturing device 14 when it is a predetermined distance away from the feature point set group 3, the vertical distance and the horizontal distance between two feature point sets 4 included in the feature point set group 3 and corresponding to adjacent vertex positions of the virtual rectangle are determined.
[0092] Fig. 9 1 is a diagram for explaining the positional relationship between the mobile terminal device 2 and the feature point set 4 included in the feature point set group 3 in the MR system 1 according to the embodiment of the present invention. Fig. 9 , the distance h represents the distance between the feature point sets 4 fixed on the rod-shaped body 41, and the distance w represents the distance between the feature point sets 4 fixed on the upper side of a pair of rod-shaped bodies 41, or the distance between the feature point sets 4 fixed on the lower side of the rod-shaped body 41. The distance d represents the distance from the feature point set group 3 to the shooting device 14 (mobile terminal device 2). The distance between the feature point sets 4 is, for example, the distance between the centers of the feature point sets 4, and the distance relative to the feature point set group 3 is, for example, the distance relative to the virtual rectangle (plane) formed by the feature point sets 4 included in the feature point set group 3. For example, the distance d is the distance from the user to the feature point set group 3 recommended by the MR system 1 to the user who wants to have an MR experience.
[0093] In one example, when the vertical viewing angle of the camera 14 is α degrees and the horizontal viewing angle of the camera 14 is β degrees, the maximum value h of the distance h is determined by the following formulas: max and the maximum value w of the distance w max .
[0094]
[0095]
[0096] By arranging each feature point set 4 so that h≤h max And w≤w max , when the photographing device 14 is away from the feature point set group 3 by a distance d, the photographing device 14 becomes able to photograph the feature point set 4 included in the feature point set group 3 in a single photographed image. In order to enable the position determination unit 23 to calculate the position more accurately, it is preferable to make the distance between the feature point sets 4 larger, and therefore it is preferable to arrange each feature point set 4 so that h=h max And w=w max In one example, with d=2m, α=63 degrees, and β=50 degrees, h=1.2m and w=0.9m.
[0097] Next, we will use Fig.10 The information processing in the MR system 1 according to the embodiment of the present invention is described with reference to the flowchart shown in FIG. Fig.10The information processing indicated in is realized by causing the mobile terminal device 2 to execute a program. In this processing, a user having the mobile terminal device 2 is present in the real space 50, and an application including the function of the application unit 21 is started by the user. In one example, when the application is running, the mobile terminal device 2 periodically executes the processing in the flowchart.
[0098] In step S101, the mobile terminal device determines whether three or more feature point sets 4 can be recognized from the captured image captured by the capturing device 14 of the mobile terminal device 2. In the case where three or more feature point sets 4 are recognized, the process proceeds to step S102, and in the case where less than three feature point sets 4 are recognized, the process is terminated.
[0099] Next, in step S102, the mobile terminal device 2 obtains arrangement position information related to the arrangement of each feature point set 4 by using each ID of the identified feature point set 4 and the feature point set position data stored in the storage device 15. The mobile terminal device 2 determines relative position information indicating the position of the mobile terminal device 2 relative to each feature point set 4 based on the shape and size of the identified feature point set 4.
[0100] Next, in step S103 , the mobile terminal device 2 determines a virtual camera viewpoint position based on each of the arrangement position information and the relative position information for each of the identified feature point sets 4 .
[0101] Next, in step S104, the mobile terminal device 2 determines whether the second virtual object 62 exists within the virtual camera field of view. In the case where the second virtual object 62 exists, the process proceeds to step S105, and in the case where the second virtual object 62 does not exist, the process terminates. In step S105, based on the data of the first virtual object 61, the data of the second virtual object 62, and the determined virtual camera viewpoint position, the mobile terminal device 2 generates a mixed reality image in which the image of the second virtual object 62 according to the viewpoint position is superimposed on the captured image of the real space 50. When the process ends, the mobile terminal device 2 directly displays the captured image of the real space 50 as a mixed reality image.
[0102] Next, the operation and advantages of the MR system 1 (mobile terminal device 2) according to an embodiment of the present invention will be described. In the present embodiment, the MR system 1 includes a plurality of feature point sets 4 in a prescribed real space 50, and each feature point set 4 is arranged at four vertex positions of a virtual rectangle in the real space 50 in a manner that is away from each other by a predefined distance in the vertical direction and away from each other by a predefined distance in the horizontal direction. In addition, the storage device 15 stores three-dimensional space data for representing a real object 51 existing in the real space 50 and defining a virtual space 60, and stores arrangement position information of each feature point set 4, which is associated with the three-dimensional space data.
[0103] By identifying the feature point sets 4, the mobile terminal device 2 obtains the arrangement position information of each identified feature point set 4. On the other hand, the mobile terminal device 2 determines the relative position information (6DoF) indicating the position of the mobile terminal device 2 relative to each feature point set 4 based on the shape and size of each identified feature point set 4 in the captured image. The mobile terminal device 2 determines the camera viewpoint position (6DoF) based on the arrangement position information and relative position information of each identified feature point set 4, and generates a mixed reality image according to the virtual camera viewpoint position related to the captured image of the real space 50.
[0104] As described above, in the MR system 1, the position of the real object 51 is associated with the position of the feature point set 4 by using the coordinates of the three-dimensional space data (virtual space 60), so that by identifying the feature point set 4, the mobile terminal device 2 can obtain the arrangement position information of the identified feature point set 4 in the virtual space 60. In addition, in the present embodiment, by using the arrangement position information and the relative position information of the mobile terminal device 2 relative to the feature point set 4, it is possible to determine the virtual camera viewpoint position (6DoF) corresponding to the shooting device 14 (self-position) being operated by the user.
[0105] In addition, in the present embodiment, after calculating each tentative virtual camera viewpoint position (6DoF) based on each feature point set 4, the mobile terminal device 2 determines the virtual camera viewpoint position based on the calculated 6DoF values. The accuracy of the virtual camera viewpoint position is very important in generating a mixed reality image. By using this configuration, in the present embodiment, it becomes possible to determine the virtual camera viewpoint position more accurately. In addition, in the present embodiment, the mobile terminal device 2 determines the virtual camera viewpoint position by further using the arrangement relationship of the known feature point set group 3. By doing so, it becomes possible to determine the virtual camera viewpoint position more accurately. In this case, in order to perform the operation more accurately and faster, when the mobile terminal device 2 determines the virtual camera viewpoint position, it is preferred that the adjacent feature point sets 4 constituting the feature point set group 3 are substantially parallel in the vertical direction (vertical direction) and the horizontal direction.
[0106] Furthermore, in the present embodiment, when generating a mixed reality image, the mobile terminal device 2 performs a drawing process for the first virtual object 61 as an invisible mesh, and performs a normal drawing process for the second virtual object 62. By using this configuration, in the present embodiment, it becomes possible to achieve a higher geometric matching (front / back relationship, occlusion, etc.) with the real object 51 when displaying the second virtual object 62. When applying a sense of reality to the second virtual object 62, the geometric matching in the mixed reality image is extremely important.
[0107] Unless explicitly stated otherwise, the above-described operations and advantages similarly apply to other embodiments or other examples.
[0108] In other embodiments of the present invention, the present invention may also be a program for implementing the functions of the above-mentioned embodiments of the present invention or the information processing shown in the flowchart, or a computer-readable storage medium storing the program. In yet another embodiment of the present invention, the present invention may also be a method for implementing the functions of the above-mentioned embodiments of the present invention or the information processing shown in the flowchart. In yet another embodiment of the present invention, the present invention may also be a server capable of providing a computer with a program for implementing the functions of the above-mentioned embodiments of the present invention or the information processing shown in the flowchart. In yet another embodiment of the present invention, the present invention may also be a virtual machine for implementing the functions of the above-mentioned embodiments of the present invention or the information processing shown in the flowchart. In yet another embodiment of the present invention, the present invention may also be an arrangement structure of a feature point set 4 in the MR system 1.
[0109] Modifications of the embodiments of the present invention will be described below. The modifications described below can be appropriately combined and applied to any embodiment as long as no contradiction occurs.
[0110] In one variation, the feature point set 4 is a two-dimensional code or includes a two-dimensional code. For example, the feature point set 4 is a QR code (registered trademark) or a data matrix. In yet another variation, the two-dimensional code used as the feature point set 4 includes an identifier for indicating latitude, longitude, angle, etc. as identifiable information. In this case, the mobile terminal device 2 that has recognized the two-dimensional code can directly obtain the layout position information of the feature point set 4 from the obtained identifiable information. In this case, the storage device 15 does not need to store the feature point set position data.
[0111] In a variation, the feature point set 4 is a known AR marker.
[0112] In a modified example, posters, paintings, pictures, etc. having spatial distribution of different feature points that are respectively identifiable are used as feature point sets 4. The recognition unit 22 and the position determination unit 23 use the known method using local features as described above (non-patent document 2 and non-patent document 3). In this case, the local features are used as identifiable information. In this modified example, the storage device 15 stores the template image of each feature point set 4 in association with the ID of the feature point set 4. The recognition unit 22 recognizes the ID of each feature point set 4 by extracting the local features from the feature point set 4 photographed by the photographing device 14 and comparing the local features with the template image of each feature point set 4. The position determination unit 23 obtains the arrangement position information related to the arrangement of the feature point set 4 by using the ID of the identified feature point set 4 and the feature point set position data stored in the storage device 15. In addition, the position determination unit 23 determines the relative position information of the mobile terminal device 2 with respect to the feature point set 4 by using the local features to derive the projection (such as rotation, enlargement / reduction and deformation) between the template image and the image photographed by the photographing device 14. For each identified feature point set 4 , the position determination unit 23 obtains arrangement position information and determines relative position information, and determines a virtual camera viewpoint position based on the arrangement position information and the relative position information.
[0113] In a modified example, the feature point set group 3 includes three feature point sets 4 arranged to have a predefined positional relationship. For example, two feature point sets in the feature point set 4 are fixedly attached to one of a pair of rod-shaped bodies 41 installed in the real space 50 in an isolated manner in the horizontal direction, and one feature point set in the feature point set 4 is fixedly attached to the other rod-shaped body 41. In the present embodiment, by including three feature point sets 4 having a predefined positional relationship in the feature point set group 3, the position determination unit 23 becomes able to determine the virtual camera viewpoint position by using the arrangement relationship of the known feature point set groups 3. In this case, preferably, the three feature point sets 4 are arranged to have a positional relationship corresponding to the positions of three of the four vertices of the virtual rectangle in the real space 50.
[0114] In a modified example, the MR system 1 includes a plurality of feature point sets 4, each of which is fixedly attached to each of three or more rod-shaped bodies 41 installed in a horizontally isolated manner in the real space 50. The plurality of feature point sets 4 are fixedly attached to each of the rod-shaped bodies 41 in a vertically isolated manner. However, some of the rod-shaped bodies 41 may have only one feature point set of the feature point sets 4 attached.
[0115] In one variation, each feature point set 4 is fixedly attached to a transmissive member 42 formed of a light-transmitting material. The transmissive member 42 is, for example, glass or resin that transmits almost the entire visible light range, and may be part of a structure such as a window glass, or may be portable (such as a glass plate). Fig.11 is a diagram showing a feature point cluster 3 fixedly attached to a window glass.
[0116] In one modification, each feature point set 4 is fixedly attached by using a line. However, the feature point set 4 may be fixed in the real space 50 by using a member having a function equivalent to a line.
[0117] In one modification, the MR system 1 includes a feature point set group 3 having five or more feature point sets 4. For example, the feature point set group 3 is composed of 20 feature point sets 4. This modification assumes a case where the imaging device 14 cannot capture all feature point sets 4 included in the feature point set group 3 in a single captured image.
[0118] The recognition unit 22 recognizes each feature point set 4 included in the image captured by the camera 14 at a prescribed time interval. The position determination unit 23 determines the camera viewpoint position (6DoF) at a prescribed time interval based on the arrangement position information and relative position information of each feature point set 4 that has been identified. Based on the data of the first virtual object 61 and the data of the second virtual object 62 and the virtual camera viewpoint position determined by the position determination unit 23 at a prescribed time interval, the image generation unit 24 generates a mixed reality image according to the virtual camera viewpoint position related to the captured image of the real space 50.
[0119] As described above, after the camera 14 identifies a plurality of feature point sets 4, when the shooting position of the camera 14 is moved by the movement of the user, the mobile terminal device 2 identifies a plurality of other feature point sets 4. By using this configuration, as the user moves in the real space 50, the mobile terminal device 2 becomes able to perform a "roaming process" for performing calibration while sequentially switching between the plurality of feature point sets 4. Therefore, in this modification, even in the case where the plurality of feature point sets 4 shot by the camera 14 change over time, the mobile terminal device 2 is able to continuously calculate its own position (6DoF). This makes it possible to continuously provide an MR experience to a user moving in the real space 50.
[0120] In this modification, the position determination unit 23 may determine the camera viewpoint position (6DoF) based on the arrangement position information and the relative position information of each feature point set in some feature point sets 4 of the identified feature point sets 4. For example, the camera viewpoint position (6DoF) may be determined based on the arrangement position information and the relative position information of each feature point set 4 having a size greater than or equal to a prescribed size in the captured image.
[0121] In a variation, if Fig.11 As shown in , the MR system 1 includes a plurality of feature point set groups 3, and the plurality of feature point set groups 3 include a first feature point set group 3a, a second feature point set group 3b, a third feature point set group 3c, and a fourth feature point set group 3d. In this variant, although each feature point set group 3 includes mutually different feature point sets 4, the plurality of feature point set groups 3 include the same feature point set 4. For example, the first feature point set group 3a includes the feature point set 4 included in the second feature point set group 3b and the feature point set 4 not included in the second feature point set group 3b.
[0122] The recognition unit 22 recognizes each feature point set 4 included in the image captured by the capturing device 14 at a prescribed time interval. The application unit 21 selects one feature point set group 3 among the identified feature point set groups 3 as a reference feature point set group 3. The position determination unit 23 determines relative position information for each identified feature point set 4 included in the reference feature point set group 3, and then determines a first virtual rectangle formed by the identified feature point sets 4. On the other hand, the position determination unit 23 obtains layout position information for each identified feature point set 4, and then determines a second virtual rectangle formed by the obtained layout position information. The position determination unit 23 determines the virtual camera viewpoint position by using the layout position information of the second virtual rectangle and the shape and size of the first virtual rectangle relative to the second virtual rectangle.
[0123] In one example, each time the recognition unit 22 recognizes each feature point set 4, the application unit 21 selects a reference feature point set group 3. In one example, the application unit 21 selects a feature point set group 3 having more feature point sets 4 included in the captured image as the reference feature point set group 3. In one example, instead of or in addition to the above selection method, the application unit 21 further selects a feature point set group 3 having more feature point sets 4 included in the central part of the captured image as the reference feature point set group 3. In one example, instead of or in addition to the above selection method, the application unit 21 further selects a feature point set group 3 having a larger ratio of feature point sets 4 included in the captured image as the reference feature point set group 3.
[0124] In this way, after the photographing device 14 identifies the feature point set 4 included in the reference feature point set group 3, when the photographing position of the photographing device 14 is moved by the movement of the user, the mobile terminal device 2 identifies the feature point set 4 included in other reference feature point set groups 3. By using this configuration, as the user moves in the real space 50, the mobile terminal device 2 becomes able to perform a "roaming process" for performing calibration while sequentially switching the feature point set groups 3. Therefore, in this variant, even if the multiple feature point sets 4 photographed by the photographing device 14 change over time, the mobile terminal device 2 can continuously calculate its own position (6DoF). This makes it possible to continuously provide an MR experience to the user moving in the real space 50.
[0125] Note that one feature point set group 3 may include six feature point sets 4 or may include nine feature point sets 4 .
[0126] In a modified example, the mobile terminal device 2 (position determination unit 23) fixes the virtual camera viewpoint position to a prescribed position (e.g., predefined 6DoF) in the virtual space 60, and dynamically determines the position of the coordinates of the three-dimensional space data. Instead of determining the camera viewpoint position, the mobile terminal device 2 determines the three-dimensional position information of the first virtual object 61. For example, the mobile terminal device 2 determines the coordinates of the three-dimensional space data. Accordingly, the coordinates of the feature point set 4 and the coordinates of the second virtual object 62 are changed.
[0127] In one modification, quaternion values are used as the arrangement position information (arrangement position and posture information) or the relative position information (relative position and posture information) instead of the 6DoF values.
[0128] In a variation, if Fig.12 As shown in , the MR system 1 is provided with a server 6 and a plurality of mobile terminal devices 2, and the mobile terminal devices 2 and the server 6 are connected to a network 8 such as the Internet and can communicate with each other. The server 6 is provided with a configuration provided in a general server. For example, the server 6 is provided with a processor such as a CPU for controlling each component, a storage element such as a ROM or RAM, a storage device such as a hard disk drive, a display device such as a display, an input device such as a mouse, a keyboard or a touch screen, and a communication device such as a network board. These constituent elements are connected via a bus.
[0129] The storage device provided in the server 6 stores the data (three-dimensional space data) of the first virtual object 61 and the data of the second virtual object 62. The storage device 15 of the mobile terminal device 2 does not need to store the data stored in the server 6. For example, the server 6 stores the three-dimensional space data together with a program for using the three-dimensional space data as a three-dimensional space database, and transmits the three-dimensional space data to the mobile terminal device 2 in response to a query from the mobile terminal device 2. The storage device provided in the server 6 can store the feature point set position data on behalf of the storage device 15 of the mobile terminal device 2.
[0130] In yet another variation, the operation of the mobile terminal device 2 of the MR system 1 is implemented by a Web application provided by the server 6. For example, the server 6 is provided with all the functions of the application unit 21 and stores various data used in the operation of the application unit 21. In this case, the server 6 provides the mobile terminal device 2 with a Web application corresponding to the application unit 21. When the Web application installed in the mobile terminal device 2 is started, the mobile terminal device 2 accesses the Web application on the server 6. The mobile terminal device 2 implements the functions of the application unit 21 on the mobile terminal device 2 by communicating with the server 6.
[0131] Unless any contradiction occurs in the process or operation (such as a contradiction that a step involves the use of data that is not yet available in the step, etc.), the process and operation can be freely modified in the above process or operation. In addition, the above embodiments are examples for explaining the present invention, and the present invention is not limited to these embodiments. The present invention can be implemented in various forms without departing from its purpose.
[0132] {List of Reference Numerals}
[0133] 1 Mixed Reality System
[0134] 2 Mobile terminal devices
[0135] 3 Feature point clusters
[0136] 4 Feature point set
[0137] 6 Server
[0138] 8 Network
[0139] 11 Processor
[0140] 12 Input Devices
[0141] 13 Display device
[0142] 14. Filming Equipment
[0143] 15 Storage Device
[0144] 16 Communication devices
[0145] 17 Bus
[0146] 21 Application Units
[0147] 22 Identification Unit
[0148] 23 Position determination unit
[0149] 24 Image generation unit
[0150] 25 Image display unit
[0151] 41 rod-shaped body
[0152] 42 Transmission components
[0153] 50 Real Space
[0154] 51 Real Objects
[0155] 60 Virtual Space
[0156] 61 First Virtual Object
[0157] 62 Second virtual object
Claims
1. A mixed reality system for displaying a mixed reality image on a display, the display being used to display virtual objects to a user existing in a prescribed real space, in which images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, the mixed reality system comprising a mobile terminal device having the display and a capturing device for capturing the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and The mobile terminal device performs the following operations: storing pre-acquired data of a first virtual object, data of a second virtual object in a virtual space that does not correspond to a real object, and storing arrangement position and posture information in the virtual space for arranging each feature point set in the prescribed real space, wherein the first virtual object corresponds to a real object existing in the prescribed real space and defines the virtual space, identifying each set of feature points captured by the camera, determining a viewpoint position of a virtual camera in a virtual space corresponding to the position and shooting direction of the shooting device in a real space based on the arrangement position and posture information for each feature point set and the relative position and posture information of the mobile terminal device relative to each feature point set, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the feature point sets in the identified feature point set, and the relative position and posture information is determined from the shape and size of the feature point set; and generating a mixed reality image in which an image of the second virtual object corresponding to the viewpoint position is superimposed on a captured image of a real space based on the data of the first virtual object, the data of the second virtual object and the viewpoint position, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
2. The mixed reality system according to claim 1, wherein: When the first virtual object and the second virtual object overlap with each other in the field of view of the virtual camera when generating the mixed reality image, if the depth distance of the second virtual object relative to the viewpoint position is greater than the depth distance of the first virtual object relative to the viewpoint position, the mobile terminal device generates a mixed reality image superimposed with an image of the second virtual object with the overlapping portion removed.
3. The mixed reality system according to claim 1, wherein: The mixed reality system includes five or more feature point sets.
4. The mixed reality system according to claim 1, wherein: The mixed reality system includes four feature point sets arranged in the specified real space, each feature point set has a positional relationship corresponding to the four vertex positions of a virtual rectangle in the real space, and each feature point set is arranged to be away from each other by a predefined distance in the vertical direction and away from each other by a predefined distance in the horizontal direction.
5. The mixed reality system according to claim 4, wherein: One or more of the feature point sets are fixedly attached to each of a plurality of rod-shaped bodies installed in a manner isolated in a horizontal direction.
6. The mixed reality system according to any one of claims 1 to 4, wherein: Each characteristic point set is fixedly attached to a transmissive member formed of a light-transmitting material.
7. The mixed reality system according to any one of claims 1 to 4, wherein: Each set of feature points is fixedly attached by using a line.
8. The mixed reality system according to any one of claims 1 to 4, wherein: The feature point set is a poster, a painting, a picture, an ArUco marker, an AR marker, a QR code or a data matrix.
9. A computer-readable medium storing a program executed by a mobile terminal device in a mixed reality system, the mixed reality system being used to display a mixed reality image on a display, the display being used to display virtual objects to a user existing in a specified real space, in which images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, the mixed reality system comprising the mobile terminal device having the display and a capturing device for capturing the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and The program causes the mobile terminal device to execute the following steps: identifying each set of feature points captured by the camera, determining a viewpoint position of a virtual camera in a virtual space corresponding to a position and a shooting direction of the shooting device in a real space based on arrangement position and posture information in a virtual space where each feature point set is arranged, and relative position and posture information of the mobile terminal device relative to each feature point set, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the identified feature point sets, and the relative position and posture information is determined from a shape and a size of the feature point sets; and A mixed reality image is generated in which an image of a second virtual object corresponding to the viewpoint position is superimposed on a captured image of the real space based on pre-acquired data of a first virtual object, data of a second virtual object in the virtual space that does not correspond to a real object, and the viewpoint position, wherein the first virtual object corresponds to a real object existing in the prescribed real space and defines the virtual space, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
10. A mobile terminal device in a mixed reality system, the mixed reality system is used to display a mixed reality image on a display, the display is used to display virtual objects to a user existing in a specified real space, in which the image of the virtual object arranged in the virtual space corresponding to the real space is superimposed on the captured image of the real space, the mixed reality system comprises the mobile terminal device having the display and a shooting device for shooting the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and The mobile terminal device performs the following operations: storing pre-acquired data of a first virtual object, data of a second virtual object in a virtual space that does not correspond to a real object, and storing arrangement position and posture information in the virtual space for arranging each feature point set in the prescribed real space, wherein the first virtual object corresponds to a real object existing in the prescribed real space and defines the virtual space, identifying each set of feature points captured by the camera, determining a viewpoint position of a virtual camera in a virtual space corresponding to the position and shooting direction of the shooting device in a real space based on the arrangement position and posture information for each feature point set and the relative position and posture information of the mobile terminal device relative to each feature point set, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the feature point sets in the identified feature point set, and the relative position and posture information is determined from the shape and size of the feature point set; and generating a mixed reality image in which an image of the second virtual object corresponding to the viewpoint position is superimposed on a captured image of a real space based on the data of the first virtual object, the data of the second virtual object and the viewpoint position, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
11. A method performed by a mobile terminal device in a mixed reality system, the mixed reality system being used to display a mixed reality image on a display, the display being used to display virtual objects to a user existing in a specified real space, in which images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, the mixed reality system comprising the mobile terminal device having the display and a capturing device for capturing the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and The method comprises the following steps: identifying each set of feature points captured by the camera, determining a viewpoint position of a virtual camera in a virtual space corresponding to a position and a shooting direction of the shooting device in a real space based on arrangement position and posture information in a virtual space where each feature point set is arranged, and relative position and posture information of the mobile terminal device relative to each feature point set, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the identified feature point sets, and the relative position and posture information is determined from a shape and a size of the feature point sets; and A mixed reality image is generated in which an image of a second virtual object corresponding to the viewpoint position is superimposed on a captured image of the real space based on pre-acquired data of a first virtual object, data of a second virtual object in the virtual space that does not correspond to a real object, and the viewpoint position, wherein the first virtual object corresponds to a real object existing in the prescribed real space and defines the virtual space, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
12. A mixed reality system for displaying a mixed reality image on a display, the display being used to display virtual objects to a user existing in a specified real space, in which images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, the mixed reality system comprising a server and a mobile terminal device, the mobile terminal device having the display and a capturing device for capturing the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets of the plurality of feature point sets are arranged to have a predefined positional relationship; The server stores pre-acquired data of a first virtual object, data of a second virtual object in a virtual space that does not correspond to a real object, and arrangement position posture information in the virtual space for arranging each feature point set in the prescribed real space, wherein the first virtual object corresponds to a real object existing in the prescribed real space and defines the virtual space; and The mobile terminal device performs the following operations: identifying each set of feature points captured by the camera, determining a viewpoint position of a virtual camera in a virtual space corresponding to the position and shooting direction of the shooting device in a real space based on the arrangement position and posture information for each feature point set and the relative position and posture information of the mobile terminal device relative to each feature point set, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the feature point sets in the identified feature point set, and the relative position and posture information is determined from the shape and size of the feature point set; and generating a mixed reality image in which an image of the second virtual object corresponding to the viewpoint position is superimposed on a captured image of a real space based on the data of the first virtual object, the data of the second virtual object and the viewpoint position, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
13. A mixed reality system for displaying a mixed reality image on a display, the display being used to display virtual objects to a user existing in a specified real space, in which images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, the mixed reality system comprising a mobile terminal device having the display and a capturing device for capturing the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and The mobile terminal device performs the following operations: storing pre-acquired data of a first virtual object, data of a second virtual object in a virtual space that does not correspond to a real object, and storing arrangement position and posture information in the virtual space for arranging each feature point set in the prescribed real space, wherein the first virtual object corresponds to a real object existing in the prescribed real space and defines the virtual space, identifying each set of feature points captured by the camera, determining the position and posture information of the first virtual object based on the arrangement position and posture information for each feature point set, and the relative position and posture information of the mobile terminal device relative to each feature point set, wherein the arrangement position and posture information is obtained from identifiable information of a part or all of the identified feature point sets, and the relative position and posture information is determined from the shape and size of the feature point sets, and generating a mixed reality image in which an image of the second virtual object corresponding to a viewpoint position is superimposed on a captured image of a real space based on the data of the first virtual object and the data of the second virtual object, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
14. A computer-readable medium storing a program executed by a mobile terminal device in a mixed reality system, the mixed reality system being used to display a mixed reality image on a display, the display being used to display virtual objects to a user existing in a specified real space, in which images of virtual objects arranged in a virtual space corresponding to the real space are superimposed on a captured image of the real space, the mixed reality system comprising the mobile terminal device having the display and a capturing device for capturing the real space, wherein: The mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information enabling identification of each feature point set, and at least three feature point sets among the plurality of feature point sets are arranged to have a predefined positional relationship; and The program causes the mobile terminal device to execute the following steps: identifying each set of feature points captured by the camera, determining, based on arrangement position and posture information in a virtual space where each feature point set is arranged, and relative position and posture information of the mobile terminal device relative to each feature point set, position and posture information of a first virtual object obtained in advance, the arrangement position and posture information being obtained from identifiable information of a part or all of the identified feature point sets, the relative position and posture information being determined from the shape and size of the feature point sets, the first virtual object corresponding to a real object existing in the prescribed real space and defining the virtual space, and generating a mixed reality image in which an image of the second virtual object corresponding to the viewpoint position is superimposed on a captured image in the real space based on the data of the first virtual object and the data of a second virtual object in the virtual space that does not correspond to the real object, The mobile terminal device identifies each feature point set captured by the shooting device at a specified time interval, determines the viewpoint position, and generates the mixed reality image, and a part of the plurality of feature point sets is captured into a single captured image. After the photographing device identifies a part of the plurality of feature point sets, when the photographing position of the photographing device is moved by the movement of the user, the mobile terminal device identifies other feature point sets in the plurality of feature point sets.
