Alignment method of virtual space with real space

Abstract

To provide a method of aligning a virtual space with respect to a real space, configured to exert high accuracy in various environments.SOLUTION: A method of aligning a virtual space with respect to a real space includes the steps of: setting a marker along a corner or an edge in a real space; measuring the position and attitude of the marker with respect to a corner or an edge of a structure; calculating the relative position and attitude of an AR device with respect to the marker, based on image information obtained by imaging the marker through an imaging unit of the AR device; and aligning a coordinate system of a virtual space stored in a storage unit in advance with respect to a coordinate system of the real space, on the basis of the information acquired in the measurement step and the information acquired in the calculation step.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a method for aligning a virtual space with a real space. [Background technology]

[0002] Conventionally, it has been known to use AR devices that display AR (augmented reality), such as AR glasses, smartphones, and tablet terminals, to overlay virtual images (e.g., 3D-CAD data) on real-world scenery or photographed images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-115957 Summary of the Invention [Problem to be solved by the invention]

[0004] In AR devices like those mentioned above, in order to align the position of the virtual image with the real-world scenery, it is necessary to acquire information such as the position, orientation, and movement of the AR device in the real world. In particular, since it is often difficult to use GPS in fields such as civil engineering and construction, it is possible to simultaneously estimate the device's position and create an environmental map using a method called VSLAM.

[0005] However, in cases where the real-world scenery has few feature points, many similar landscape elements, or the environment is constantly changing, such as in tunnel construction, conventional methods have the problem that the accuracy of alignment between real space and virtual space is easily reduced.

[0006] Therefore, the present disclosure has been made in consideration of the above technical problems, and its purpose is to provide a method for aligning a virtual space with a real space that can demonstrate high accuracy in various environments. [Means for solving the problem]

[0007] In the present disclosure, the method for aligning a virtual space with a real space includes the steps of placing a marker along a corner or edge in the real space, measuring the position and orientation of the marker with respect to the corner or edge of a structure, calculating the relative position and orientation of the AR device with respect to the marker based on image information obtained by capturing an image of the marker with an imaging unit of the AR device, and aligning the coordinate system of the virtual space, which has been pre-stored in a memory unit, with the coordinate system of the real space based on the information obtained in the measuring step and the information obtained in the calculating step. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a method for aligning a virtual space with a real space, which can demonstrate high accuracy in various environments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of an AR system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of an AR device according to the present embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a server according to the present embodiment. [Figure 4] 10A and 10B are diagrams illustrating an example of installation of an AR marker according to the present embodiment. [Figure 5] FIG. 10 is a flowchart illustrating a method for estimating a self-position of an AR device according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating other examples of placement of AR markers according to the present embodiment. [Figure 7]FIG. 1 is a flowchart illustrating a method for aligning a real space and a virtual space according to an embodiment of the present invention. [Figure 8] FIG. 10 is a plan view showing an example of the placement of markers according to the present embodiment. [Figure 9] FIG. 10 is a flowchart illustrating a method for aligning a virtual space with a real space according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] Fig. 1 shows an example of an AR system according to an embodiment of the present invention. As shown in Fig. 1, the AR system of this example includes an AR device 10 and a server 20. The AR device 10 and the server 20 are connected via a network 30 and are capable of communicating with each other.

[0012] <ARデバイス> The AR device 10 in this example is AR glasses, which are glasses-type information processing terminals. Note that the AR device 10 is not limited to AR glasses, and may be an information processing device such as a smartphone or a tablet terminal.

[0013] 2 is a block diagram illustrating an outline of the configuration of the AR device 10. As shown in the figure, the AR device 10 includes, for example, a control unit 11, a storage unit 12, a display unit 13, a communication unit 14, an imaging unit 15, and a sensor unit 16, which are connected to each other via a bus 17.

[0014] The control unit 11 exchanges data between the various units and controls the entire AR device 10. For example, the control unit 11 is implemented by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The control unit 11 can access the storage unit 12 and can store information in the storage unit 12 and read information from the storage unit 12.

[0015] The control unit 11 acquires, generates, outputs, etc. various types of data by executing programs stored in the storage unit 12. The control unit 11 acquires information acquired from other devices such as the imaging unit 15, the sensor unit 16, and the server 20, and can estimate the position and attitude (orientation) of the AR glasses based on the acquired information and the information in the storage unit 12, etc.

[0016] The storage unit 12 stores programs, codes, etc. for performing one or more steps, and the control unit 11 controls the operation of each unit in accordance with the programs, etc. The storage unit 12 has a nonvolatile storage device which is a read-only storage area in which a system program is stored, and a volatile storage device which is a rewritable storage area used as a work area for arithmetic processing by the control unit 11. The nonvolatile storage device is realized by, for example, a ROM (Read Only Memory), a flash memory, a hard disk, etc., and the volatile storage device is realized by a RAM (Random Access Memory), a VRAM (Video Random Access Memory), etc.

[0017] The storage unit 12 may include, for example, a separable medium such as an SD card or random access memory (RAM) or an external storage device. The storage unit 12 can store various data acquired from the imaging unit 15 and the sensor unit 16. For example, image data such as still images and videos captured by a camera serving as the imaging unit 15 may be stored in an internal memory or an external memory.

[0018] The display unit 13 includes a display device such as a liquid crystal display device. The display device can be, for example, a waveguide display. An image controlled by the control unit 11 is displayed on the display device. That is, the display unit 13 can display an image generated by the control unit 11. The display device also has a transparent portion that is partially or entirely transparent or semi-transparent, and can display a virtual image superimposed on a scene in real space while viewing the scene through the transparent portion. The display unit 13 may include one or more lenses separate from the display device.

[0019] The display unit 13 displays a virtual image (3D-CAD model, CG model, etc.) corresponding to the position (imaging position) and posture (imaging direction) of the AR glasses when an image of real space (for example, a marker, etc., described later) is captured (photographed), superimposed on the scenery of real space visible through the lens of the display device. Note that the display unit 13 may generate an AR image by superimposing a virtual image on an image captured by the imaging unit 15 using the control unit 11, etc., and display the AR image.

[0020] The communication unit 14 connects the AR device 10 to a network 30 including the Internet. The communication unit 14 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0021] The imaging unit 15 includes one or more imaging devices (cameras). The control unit 11 can send image data of still images or videos captured by the imaging devices to the storage unit 12, the server 20, etc. The imaging devices can include, for example, an RGB camera, a pair of left and right fisheye cameras for VSLAM, a ToF camera (Time-of-Flight Camera), etc. Such imaging devices can also function as the sensor unit 16. For example, a ToF camera (Time-of-Flight Camera) can determine the distance to an object by measuring the time of flight of light, and is also called a ToF sensor.

[0022] The imaging unit 15 can capture continuous images over a predetermined period of time in response to a user's operation or based on preset instruction information. The imaging unit 15 can also capture discontinuous images at a single point in time in response to a user's operation or based on preset instruction information. In other words, the imaging unit 15 can capture still images and moving images.

[0023] The sensor unit 16 can be, for example, a three-axis acceleration sensor, a three-axis gyro sensor, a direction sensor, a GPS, etc. Each sensor can transmit (input) to the control unit 11 information (detection signals) such as the position, posture, speed of movement, acceleration, etc. detected by each sensor.

[0024] The AR glasses may include an input unit that accepts input of requests (instructions) from a user or the like. The input unit may be, for example, a touch panel or a mechanical switch. The input unit may also accept gaze input using eye tracking technology or voice input using a microphone or the like. The input unit may also accept input from another information processing device such as a smartphone. The AR glasses may also include an output unit such as a speaker that outputs voice or the like.

[0025] <server> The server 20 is an information processing device used by a system administrator or the like when operating and managing various services, and may be, for example, a general-purpose computer such as a workstation or personal computer, or may be logically realized using cloud computing technology.

[0026] As shown in FIG. 3, the server 20 includes, for example, a control unit 21, a storage unit 22, an output unit 23, a communication unit 24, and an input unit 25, which are connected to one another via a bus 26. The server 20 is capable of communicating with the AR glasses via the communication unit 24. When the server 20 receives various request signals from the AR glasses or the like, the control unit 21 executes processing according to a program, and transmits the processing results (for example, generated images, sounds, etc.) to the AR glasses or the like as appropriate, or stores them in the storage unit 22. Note that part of the program may be transmitted to the AR glasses and executed on the AR glasses.

[0027] The control unit 21 transfers data between each unit and controls the entire server 20, and is realized by the CPU (Central Processing Unit) and GPU (Graphics Processing Unit) executing programs stored in a specified memory (storage unit).

[0028] The storage unit 22 has a nonvolatile storage device which is a read-only storage area in which a system program is stored, and a volatile storage device which is a rewritable storage area used as a work area for arithmetic processing by the control unit 21. The nonvolatile storage device is realized by, for example, a ROM (Read Only Memory), a flash memory, or a hard disk, while the volatile storage device is realized by, for example, a RAM (Random Access Memory), a VRAM (Video Random Access Memory), or the like.

[0029] The storage unit 22 can store information about the virtual space, information about the virtual image, information about the marker, information about each coordinate system (real space coordinate system, AR device coordinate system, marker coordinate system, virtual space coordinate system), and the like.

[0030] The storage unit 22 can store information received from an AR camera, etc. The storage unit 22 may also store information stored in the storage unit 12 of the AR camera.

[0031] The output unit 23 is composed of a speaker for outputting sound, a display for displaying images, and the like.

[0032] The communication unit 24 connects the server 20 to a network 30 including the Internet. The communication unit 24 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0033] The input unit 25 can be a keyboard, a mouse, a touch panel, or the like.

[0034] FIG. 4 is a diagram showing a method for improving the accuracy of self-position estimation of the AR device 10 by using AR glasses as the AR device 10 and markers 40 installed on the wall surface of a structure 50 on-site.

[0035] <marker> Marker 40 may be, for example, AprilTag, but is not limited to this, and may be other AR markers (AR tags) such as ARTag, ARToolkit, ARToolkitPlus, RUNE-Tag, reacTIVison, etc. Marker 40 is, for example, square as a whole. Marker 40 has a matrix structure in which multiple square pixels are arranged two-dimensionally, and each pixel is displayed in white or black. The shape and drawn figure of marker 40 are not limited to this.

[0036] When AprilTag is used as the markers 40, for example, the 36h11 mark series is made up of 587 markers. By simply capturing an image of any marker 40 with the camera of the AR device 10, it is possible to obtain the ID (identification information) unique to each marker 40. When capturing an image of any marker 40 with the camera of the AR device 10, the marker 40 may be captured from the front or from an oblique angle, and the control unit processes the image to obtain the identification information, position, and orientation information unique to each marker 40. Images of each marker 40 and ID information corresponding to each image are associated with each other in advance and stored in the storage unit 22 (or storage unit 12).

[0037] Specifically, for example, when image data captured by the camera of the AR glasses is sent to the server 20, the control unit 21 of the server 20 refers to the data stored in the storage unit 22 and sends ID information corresponding to that marker 40 to the AR glasses. This allows the control unit 11 of the AR glasses to obtain the ID corresponding to that marker 40. For example, information such as the position and orientation in real space can be associated with this ID information and stored in the storage unit 22. This makes it possible to accurately ascertain information such as the position and orientation of the marker in real space as well as the ID of the marker 40 simply by capturing an image of the marker 40 with the camera.

[0038] Such information about the marker 40 can be displayed on the display unit 13 of the AR glasses. The user can check information about the marker 40, such as its ID, position, and attitude, on the display unit 13 simply by capturing an image of the marker 40 with the camera of the AR glasses. Note that the process of acquiring the ID of the marker 40 based on the image of the marker 40 captured by the camera may be performed by the AR camera alone (without communicating with the server 20). In this case, information about the marker 40 is stored in advance in the storage unit 12 of the AR camera.

[0039] <Coordinate system> 4, a coordinate system specific to the AR glasses (AR device coordinate system) is set for the AR glasses. Also, a coordinate system specific to the marker 40 (marker coordinate system) is set for the marker 40. Furthermore, a real space coordinate system (world coordinate system) is set for the real space, and a virtual space coordinate system (virtual space coordinate system) is set for the virtual model.

[0040] Each coordinate system is, for example, a three-dimensional Cartesian coordinate system, and has three orthogonal coordinate axes (x-axis, y-axis, z-axis) and an origin where the three coordinate axes intersect. Information about each coordinate system can be stored in storage units 12 and 22. That is, the origin position of each coordinate system, the orientation of the coordinate axes, and the relative positions and attitudes (orientations) between each coordinate system can be stored in storage units 12 and 22. Note that the position can be expressed by three-dimensional coordinates, and the attitude can be expressed by three-dimensional directions (the rotation angles around each of the x-axis, y-axis, and z-axis).

[0041] <Virtual data> The storage unit 22 can store information about a three-dimensional virtual space (virtual space) and information about a virtual model (a three-dimensional or two-dimensional CG model) positioned at a predetermined position and orientation in the virtual space. The virtual data includes information about a virtual space coordinate system. The virtual space data preferably corresponds to spatial data of a predetermined real space (e.g., a construction site). The position, direction, and coordinate system in the virtual space are associated with the position, direction, and coordinate system in the real space. The origin position of the virtual space coordinate system can be made to coincide with (or correspond to) a predetermined position in the real space. Furthermore, the three coordinate axes of the virtual space coordinate system can be made to coincide with (or correspond to) three coordinate axes set in the real space. Furthermore, the shape, position, and orientation of the virtual model in the virtual space correspond to, for example, the shape, position, and orientation of a building or the like installed at an actual construction site in the real space.

[0042] The virtual model can be created from, for example, 3D CAD data or BIM (Building Information Modeling) data of the object. The method for creating the virtual image and the type of image data are not particularly limited. The storage unit 12 stores position information of the virtual model relative to the origin of the virtual space coordinate system and orientation information indicating the direction in which the virtual model faces in the virtual space coordinate system. The virtual space and the placement of the virtual model are configured in advance based on design data such as BIM data. The virtual model may also be based on a floor plan (e.g., a 2D construction drawing or a detailed floor plan). The virtual model may be generated by the server 20 or by another device. The AR glasses may acquire data related to the virtual model from the server 20 in advance, or may repeatedly acquire data related to the virtual model continuously (or intermittently) while the AR glasses are in use.

[0043] The origin and each coordinate axis of the AR device coordinate system are set at a predetermined position and orientation relative to the AR glasses. The control unit 11 (or the control unit 21) can estimate the position and orientation of the AR glasses based on an image captured by the imaging unit 15 of the AR glasses, and can associate the virtual data with the real space coordinate system.

[0044] AR glasses are devices that display a virtual image of an object superimposed on a real-space landscape (or an image of the real world) based on an image (video) of real space captured by an imaging unit 15. AR glasses are used as portable devices, such as worn on the head by users such as workers and supervisors at civil engineering, construction, railway, and transportation sites. Virtual images of components, buildings, etc. to be installed at the intended installation positions and in the intended installation orientations (postures) are superimposed on the real-space landscape (or an image of the real world) at the site. The virtual images are generated in advance using 3D-CAD or the like and stored in the storage unit 12, 22 of the AR device 10 or the server 20, etc.

[0045] The AR glasses acquire information about the position and attitude (angle) of the AR glasses, and the control unit 11 acquires and references data from the memory unit 12 to generate image data corresponding to the position and attitude. The generated image data (virtual image data) is then displayed on the display unit 13. As a result, a virtual image corresponding to the position and attitude of the AR glasses is displayed on the AR glasses so as to be superimposed on the scenery of the real world. By checking the virtual image at the work site, the user can work while keeping in mind the work process or final image, allowing the work to proceed reliably while reducing work errors. Furthermore, even when giving instructions from a remote location, instructions can be given efficiently by talking while viewing a three-dimensional virtual image.

[0046] The position information and posture information (direction information) of the AR glasses can be obtained, for example, by a self-location estimation technique. Examples of self-location estimation techniques include techniques that utilize image information, such as Visual SLAM (Simultaneous Localization And Mapping) and VIO (Visual Inertial Odometry), and the AR glasses may be equipped with cameras, sensors, and the like to realize these techniques. Such self-location estimation techniques are particularly effective in non-GPS environments. The AR glasses of this example can simultaneously estimate their own position and create an environmental map (virtual space) from the video captured by the imaging unit 15 using so-called "Visual SLAM."

[0047] The AR glasses of this embodiment are eyeglass-type display devices worn on the user's head, and may include one or more lenses (integrated with or separate from the display unit 13), a frame (frame body) supporting the lenses, etc., a control unit 11, an imaging unit 15, a sensor unit 16, a display unit 13, a communication unit, a memory unit 12, an input unit, etc. Note that some components of the AR glasses (such as the control unit 11) may be separated as separate units and connected by wire or wirelessly, or the processing performed using the control unit 11 and the memory unit 12 may be executed by the server 20.

[0048] The position and orientation of the AR glasses are determined by a self-position estimation unit in the control unit 11. The AR glasses are equipped with a communication unit for connecting to an information processing device such as a server 20 via a network such as the Internet or directly. This allows various data to be transmitted and received between the AR glasses and other information processing devices equipped with a communication unit. For example, image data captured by the AR glasses, image data generated by the control unit 11, sensor data acquired by the sensor unit 16, etc. can be transmitted to the server 20 and stored and processed on the server 20 side, or data such as images transmitted from the server 20 can be received.

[0049] Note that an image captured by the imaging unit 15 of the AR device 10 may be transmitted to the server 20, and an image to be displayed on the AR device 10 may be generated in the control unit 21 of the server 20.

[0050] The control unit 11 acquires information on the position (imaging position) and posture (imaging direction) of the imaging unit 15 (camera), generates an image of a virtual model corresponding to the position and posture of the imaging unit 15 when the image was captured (based on the information in the memory unit 12), and outputs it to the display unit 13.

[0051] The control unit 11 generates an image of the virtual model when the image is captured by a virtual camera, which is assumed to be located in the virtual space, under the same conditions as when the image was captured by the imaging unit 15, and the display unit 13 displays the image so that it is superimposed on the scenery in the real space (or an image of the real space).

[0052] When displaying the virtual model on the display unit 13 of the AR glasses, not only the position and orientation of the camera but also other parameters such as the brightness of the real space may be taken into consideration. The virtual model may be made semi-transparent so that overlapping portions (hidden portions) of the virtual model image can be seen. The display unit 13 of the AR glasses sets and modifies the virtual image to be displayed in accordance with the position and orientation of the camera.

[0053] The control unit 11 can estimate the movement state (presence or absence of movement, movement direction, movement speed, etc.) of the imaging device of the AR glasses based on information from the imaging unit 15 or the sensor unit 16. The control unit 11 calculates the position and orientation of the camera in the virtual space (the position and orientation if the camera were in the virtual space) according to the movement state information. The display unit 13 overlays a virtual model on the image based on the corrected position and orientation. By continuously performing this process, a virtual image that follows the movement of the AR glasses in accordance with the user's movement can be displayed on the display unit 13.

[0054] The display unit 13 generates an image to be displayed and outputs it to the display each time it inputs a captured image (continuously or intermittently) or receives information indicating the movement state from a sensor unit or the like. That is, the display unit 13 displays a virtual model superimposed on a landscape or image in real space so as to follow the movement state of the camera. This function of the display unit 13 can be realized by a known AR method (for example, the tracking algorithm or SLAM function of ARToolKit).

[0055] Hereinafter, a method for calibrating (correcting) the self-position and attitude using the marker 40 will be described with reference to FIGS.

[0056] First, as shown in Fig. 4, the marker 40 is positioned and installed in advance at a predetermined position on a construction site or the like (step S1). The position and attitude (orientation) of the marker 40 in the real space coordinate system are stored in advance in a storage unit (12, 22) of at least one of the AR glasses or the server 20. In the example of Fig. 4, the marker 40 is installed on a vertical wall surface of the structure 50, but it may also be installed on a horizontal surface or the like. Furthermore, the marker 40 is attached so that two sides are parallel to the top edge (the horizontal edge located at the top end) and the side edge (the vertical edge located at the side end) of the wall surface of the structure 50, but this is not limited thereto.

[0057] The user adjusts the orientation of the camera of the AR glasses so that the marker 40 is located within the imaging range of the camera, and captures an image of the marker 40 with the camera (step S2).

[0058] The control unit (11, 21) calculates the three-dimensional position and orientation of the marker 40 on the AR device coordinate system (camera coordinate system) based on the image data of the captured marker 40. In doing so, the position and orientation of the marker 40 are determined using a point constituting one of the corners of the square marker 40 and two sides (edges) perpendicular to the corner as reference elements (reference point, reference line). The control unit (11, 21) calculates the three-dimensional relative positional relationship (coordinate transformation matrix (R, T)) between the AR glasses coordinate system and the marker coordinate system.

[0059] The control unit (11, 21) also calculates the position and orientation of the AR glasses on the marker coordinate system by performing inverse calculation using the coordinate transformation matrix (R, T) based on the position and orientation of the marker 40 on the AR device coordinate system. As a result, the control unit (11, 21) calculates the relative position and orientation of the AR glasses with respect to the marker 40 (step S3).

[0060] The control unit (11, 21) acquires identification information, position, and orientation information of the marker 40 in the real space coordinate system from the storage unit (12, 22) based on image data of the captured marker 40. In other words, the control unit (11, 21) acquires the three-dimensional relative positional relationship (coordinate transformation matrix (R0, T0)) between the marker coordinate system and the real space coordinate system from the storage unit (12, 22). The position and orientation of the marker 40 on the real space coordinate system and the relative three-dimensional positional relationship (position and orientation) between the real space coordinate system and the marker coordinate system are stored in advance in the storage unit (12, 22). The control unit (11, 21) solves a so-called PnP problem (Perspective n-Point) that determines an image of an object whose position and shape are known, and the position and orientation of the camera that captured the image, thereby making it possible to estimate the position coordinates (image capture position coordinates) and orientation of the camera and the AR glasses including the camera.

[0061] The control unit (11, 21) calculates the position and orientation of the AR glasses on the real space coordinate system based on information on the relative position and orientation of the AR glasses with respect to the marker 40 and the relative three-dimensional positional relationship between the real space coordinate system and the marker coordinate system (i.e., by performing inverse calculation using the coordinate transformation matrix (R0, T0)) (step S4).

[0062] This method allows the AR glasses to estimate their own position and orientation in the real-space coordinate system with high accuracy.

[0063] A method for aligning the virtual space with the real space will be described below with reference to Figures 6 and 7. More specifically, a method for initial setup when using an AR device on-site will be described.

[0064] The method for aligning a virtual space with a real space according to this embodiment includes the steps of: placing a marker along a corner or edge of the real space; measuring the position and orientation of the marker with respect to the corner or edge of a structure; calculating the relative position and orientation of the AR device with respect to the marker based on image information obtained by capturing an image of the marker with an imaging unit of the AR device; and aligning the coordinate system of the virtual space, which is stored in advance in a storage unit, with the coordinate system of the real space based on the information obtained in the measuring step and the information obtained in the calculating step. With this configuration, it is possible to align the virtual space with the real space with high accuracy in any environment, i.e., in a variety of environments.

[0065] Specifically, for example, when using the AR glasses for the first time on-site, the user performs initial setup. First, an application stored in the AR glasses is started by turning on the power switch of the AR glasses, for example. Then, as the initial setup, an initial alignment is performed between a virtual space coordinate system (virtual world) in which a three-dimensional virtual image is set and a real space coordinate system (real world). Information about the virtual space coordinate system in which the virtual image is set is stored in the storage units 12 and 22. The information about the virtual space coordinate system includes information about the virtual coordinate system, the virtual model, and information about markers in the virtual space.

[0066] At the site, for example, near the entrance to a space where a building, structure, or the like will be installed, a marker 40 is installed near a position that is easy to refer to in a three-dimensional design drawing (such as 3D-CAD) (step S11). An example of an easy-to-reference position is a corner (an outer corner or an inner corner) or edge (such as a straight edge) of a building. When the control unit performs self-location estimation and mapping (map formation) using VSLAM or the like, an edge (edge) where two adjacent planes intersect or a corner where three planes intersect can be used as reference point P. When installing the marker, it can be attached using adhesive means or the like, allowing the marker to be detachably fixed to the object.

[0067] 6, a corner (inner corner) where three mutually orthogonal surfaces (a horizontal floor surface and two vertical wall surfaces) in a building intersect is set as the reference point P. A first marker 40a is placed on the floor surface, and a second marker 40b is placed on the wall surface (surface of a side wall).

[0068] When installing the markers 40, it is preferable to install multiple markers 40 on different planes (wall surface, floor surface). This allows more information to be referenced than when using only one marker 40 for alignment, thereby improving the accuracy of alignment. Note that the positions where the markers 40 are installed are preferably a vertical wall surface or a horizontal floor surface, but are not limited to this. Furthermore, from the perspective of improving the accuracy of alignment, it is preferable to install the markers 40 on planes that are perpendicular to each other. This allows the accuracy of alignment to be improved compared to when the markers are installed on planes that extend obliquely relative to each other.

[0069] It is also preferable to place markers 40 on each of the multiple wall surfaces that form the corner that is the reference point P. This allows multiple markers to be placed at different angles around the corner that is the reference point P, further improving the accuracy of alignment.

[0070] It is also preferable to place multiple markers 40 within an area that can be simultaneously imaged by the imaging unit 15 of the AR device 10 (i.e., at relatively close positions). In the example of FIG. 6, a first marker 40a and a second marker 40b are placed on planes (floor and wall) that are perpendicular to each other. As in this example, it is preferable to place the two markers 40 so that they sandwich an edge (a linear edge where two planes intersect) on both sides. For example, it is preferable that the two markers 40 are positioned so that they partially or entirely overlap in the direction along the edge between them. This allows the two markers 40 to be placed in close positions.

[0071] The markers 40 are preferably installed along corners or edges of a building. More specifically, the markers 40 are preferably installed so that two perpendicular sides of a wall that forms a corner of the interior wall of a building and two sides that form a corner of the square marker are parallel to each other. Alternatively, the markers 40 are preferably placed so that one side of the square marker 40 is parallel to one edge of a structure such as a building. This makes it easy to install the marker 40 in an appropriate position and orientation, and also facilitates positioning the virtual space relative to the real space. The markers may be installed at a predetermined position (latitude and longitude) and in a predetermined orientation (e.g., one side of the marker is parallel to any of the directions of east, west, north, and south) on the ground. The markers 40 can also be placed offset along a plane relative to a reference corner or edge by using a method such as marking.

[0072] Using a measuring device such as a laser positioning device, data (R0, T0) indicating the relative deviation of each marker 40 from the reference point P and coordinate axes (azimuth axes) of the 3D-CAD design drawing (i.e., the relative position and orientation of the marker 40) is measured (step S12). Causes of the deviation include, for example, tolerances (allowable errors) between the design drawing and the actual building, and marker installation errors. Errors in marker position and orientation recognition by the camera of the AR glasses may also be a factor. Specifically, in real space, the positions and orientations of the markers 40 are measured (actually measured) using the corners (reference points P) and edges of the building as reference points. This allows the relationship (position and orientation) between the real-space coordinate system and each marker coordinate system to be calculated. Information on the position and orientation of the marker 40 in real space is stored in the memory unit 22 of the server 20. The server 20 transmits this information to the AR glasses. Note that data indicating the deviation (coordinate transformation matrix (R0, T0)) may be stored in the memory unit 12 of the AR glasses. The measuring device is not limited to a laser positioning device, but may be any device capable of measuring the distance, orientation, etc. between two specific points in real space. For example, it may be a level or a telescope that measures angles and distances from a long distance. Furthermore, a measuring method using marking, including an automatic marking device or a laser marking device, may also be used. An automatic marking device can measure the relative position of the device with respect to a reference point to determine the current position, or draw a line on the floor at the position of the reference line. The measuring device may, for example, calculate the shape and dimensions of the floor, wall, and reference point from an image captured by a camera. Alternatively, the measuring device may obtain the dimensions of the floor using a laser distance measuring device.

[0073] Then, at the site, the installed markers 40 (40a, 40b) are captured by the imaging unit 15 of the AR glasses (step S13). The control unit of the AR glasses automatically detects the markers 40. The control unit calculates the relative position and orientation of the AR glasses with respect to each marker 40 (40a, 40b) (step S14). Then, using information on the relative position and orientation of the AR glasses with respect to each marker 40 (40a, 40b) and the data (R0, T0) indicating the deviation, an initial alignment of the virtual space with the real space is realized (step S15). In other words, the real space coordinate system and the virtual space coordinate system are associated. Preferably, the origin position and the directions of the three coordinate axes of the real space coordinate system are aligned with the origin position and the directions of the three coordinate axes of the virtual space coordinate system.

[0074] As in this example, it is preferable to place the markers 40 on two orthogonal planes, which can further improve the accuracy of alignment. Also, by placing the markers 40 on three planes that are orthogonal to each other, it is possible to further improve the accuracy of alignment.

[0075] The technology disclosed herein is particularly effective in places with few feature points, such as tunnel construction, places with many similar landscapes, or situations where the environment is constantly changing. Fields that face such challenges include civil engineering, architecture, railways, and transportation.

[0076] Hereinafter, a method for calibrating (correcting) the self-position and attitude using a plurality of markers 40 will be described with reference to FIGS. 8 and 9. FIG. 8 is a plan view showing a case where a plurality of markers are installed at a predetermined fixed interval from the entrance inside a building in real space. The markers may be installed on the floor or wall, or at other positions. In the example of FIG. 8, six markers are installed, but this is not limiting. As described above, it is preferable to install each marker 40 along a corner or edge of the current space-time that serves as a reference point.

[0077] The marker placement method according to the present embodiment includes the steps of placing a plurality of markers at a predetermined interval in a real space, storing virtual data including a plurality of virtual markers corresponding to the plurality of markers in the real space in a storage unit, measuring a deviation between the markers in image data obtained by capturing the markers in the real space with an imaging unit of an AR device and the virtual markers in virtual data displayed superimposed on the image data, and adjusting the distance between the plurality of markers according to the deviation. This configuration enables efficient placement of a plurality of markers when aligning a virtual space and a real space in an AR device using markers.

[0078] 8 and 9, a plurality of markers 40 are placed at regular intervals in real space (step S21). For example, a first marker 40 for alignment is placed near the entrance of a space such as a building or structure. The plurality of markers 40 may be the same or different types.

[0079] Next, a second marker 40 is placed according to the space so that its distance from the first marker 40 is a specific distance (for example, 10 m). Similarly, a third marker 40 is placed according to the specific distance from the second marker 40, and a fourth marker 40 is placed according to the specific distance from the third marker 40. In this way, the markers 40 are placed at equal intervals according to the space (situation) of the site. Note that the specific distance is not particularly limited and is a value that can be changed as appropriate.

[0080] Here, virtual markers are set in the virtual space (step S22). The virtual markers may be set before or after step S21 of setting the markers in real space. For example, in a virtual model of the virtual space stored in the storage unit, each marker 40 is drawn (3D modeled) and stored so that it matches the position and posture of the marker in real space. As a result, virtual markers corresponding to the markers 40 in real space are displayed as virtual images displayed on the display unit 13 of the AR glasses. Note that a virtual model including walls and floors corresponding to (having the same shape as) structures including walls and floors installed at the site in real space is stored in advance in the storage unit as virtual data positioned in the virtual space.

[0081] Then, while checking the virtual image displayed on the AR glasses, the subject moves through the space from the entrance of the site in real space, capturing images of the first marker, the second, the third, and so on, using the imaging unit of the AR glasses. During this process, the position and orientation of each virtual marker (the virtual marker displayed on the AR glasses) that is assumed to be superimposed on each marker 40 in real space is compared with the position and orientation of the marker 40 in real space. The AR glasses continuously estimate their own position and orientation using a VSLAM or the like, and display corresponding virtual images on the display unit. Then, it is determined whether the deviation between the position and orientation of the marker in real space and the position and orientation of the virtual marker exceeds a predetermined threshold (step S23). The threshold may be a value previously determined by an administrator and stored in the storage unit, or may be set by the user by inputting a numerical value via the input unit or selecting from multiple preset options. The threshold for positional deviation is expressed as a distance, such as 5 mm, and the threshold for orientation deviation is expressed as an angle, such as 5°.

[0082] The measurement of the amount of deviation can be performed, for example, by the control unit of the AR glasses or the server. When a user captures an image of a marker using the imaging unit of the AR glasses, the control unit superimposes the image captured by the imaging unit of the AR glasses on a virtual image to be displayed on the display unit of the AR glasses, and measures the deviation between the position and orientation of the marker in the captured image of real space and the position and orientation of the virtual image. More specifically, the control unit can compare the marker coordinate system of the marker in the captured image of real space with the virtual marker coordinate system of the virtual marker in virtual space. The control unit can measure the distance between the origin of the real marker coordinate system and the origin of the virtual marker coordinate system, or compare the orientation (orientation) of the coordinate axes of the real marker coordinate system and the coordinate axes of the virtual marker coordinate system and measure the difference (angle) between them. Furthermore, the control unit can compare and measure reference points, for example. For example, the control unit can measure the deviation between the marker in the real image and the marker in the virtual image by measuring the distance between specific corners of the rectangular marker 40, or the distance between the sides and the relative angle. However, the method for measuring the deviation is not limited to this.

[0083] If the deviation exceeds a threshold, the distance between the markers 40 is adjusted (step S24). For example, if the deviation between the actual second marker 40 and the second virtual marker exceeds 10 mm, the distance between the markers 40 is shortened. Specifically, if the deviation exceeds the threshold after checking the deviation with the distance between the markers 40 set to 10 m as described above, the distance between the markers is shortened by, for example, 2 m to 8 m, thereby shortening the distance from the initial distance of 10 m. In this example, the position of the second marker is first adjusted, for example, by moving the second marker so that it is closer to the first marker (placed closer). The extent to which the distance is shortened may be predetermined and stored in a memory unit, or may be determined by the user. Alternatively, the adjustment distance may be determined depending on the magnitude of the deviation. In this case, the adjustment distance may be increased if the deviation is large, and conversely, the adjustment distance may be decreased if the deviation is small. In the above example, the adjustment distance is set to 2 m. If there is almost no deviation, the distance between the markers may be increased. In this case, the extent to which the distance is increased may be determined in advance and stored in a storage unit, or may be determined by the user. Alternatively, the adjustment distance may be determined according to the magnitude of the amount of deviation.

[0084] After adjusting the position and orientation of the marker in real space, the position of the virtual marker in virtual space is changed accordingly and stored in a storage unit. At this time, the position and orientation of the marker in the current space-time may be measured using a measuring device such as a laser positioning device. This allows modeling of the virtual marker with a more accurate position and orientation.

[0085] As described above, by checking the position and orientation deviation between the marker in real space and the virtual marker in the virtual image, the accuracy of the movement detection of the AR glasses can be checked. It can also be checked whether the distance between the markers 40 is appropriate (depending on the accuracy of the AR glasses). As a result, the markers 40 can be placed with an appropriate feel, thereby improving the efficiency of use of the markers 40. In other words, it is possible to prevent the wasteful use of too many markers 40.

[0086] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0087] The devices described in this specification may be realized as a single device, or may be realized by a plurality of devices (e.g., cloud servers) some or all of which are connected via a network. For example, the control unit 21 and the storage unit 22 of the server 20 may be realized by different servers connected to each other via a network.

[0088] The series of processes performed by the apparatus described herein may be implemented using software, hardware, or a combination of software and hardware. A computer program for implementing each function of the AR device 10 and server 20 according to this embodiment may be created and installed on a PC or the like. A computer-readable recording medium storing such a computer program may also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

[0089] Furthermore, the processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may be employed, and some process steps may be omitted.

[0090] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0091] The following configurations also fall within the technical scope of the present disclosure. (Item 1) placing markers along corners or edges in real space; measuring the position and orientation of the marker relative to the corner or edge; calculating a relative position and orientation of the AR device with respect to the marker based on image information obtained by capturing an image of the marker with an imaging unit of the AR device; a step of aligning a coordinate system of the virtual space, which is stored in advance in a storage unit, with a coordinate system of the real space, based on information obtained in the measuring step and information obtained in the calculating step. (Item 2) Item 2. The method for aligning a virtual space with a real space according to item 1, wherein in the step of placing the markers, a plurality of the markers are placed on different planes. (Item 3) 3. The method for aligning a virtual space with a real space according to item 1 or 2, wherein in the step of placing the markers, a plurality of the markers are placed on planes that are orthogonal to each other. (Item 4) 4. The method for aligning a virtual space with a real space according to any one of items 1 to 3, wherein in the step of placing the markers, a plurality of the markers are placed within an area that can be simultaneously imaged by an imaging unit of the AR device. (Item 5) 6. The method for aligning a virtual space with a real space according to any one of items 1 to 5, wherein in the step of placing the marker, the marker is placed so that the edge and one side of the rectangular marker are parallel to each other. [Explanation of symbols]

[0092] 1. AR System 10 AR devices 20 servers 30 Network 40 markers

Claims

1. placing markers along edges in real space; measuring the position and orientation of the marker relative to a predetermined reference point in real space and the edge; calculating a relative position and orientation of the AR device with respect to the marker based on image information of the marker captured by an imaging unit of the AR device; and a step of aligning a coordinate system of a virtual space stored in advance in a storage unit with a coordinate system of a real space based on information obtained in the measuring step and information obtained in the calculating step, In the step of placing the markers, the markers are placed on two surfaces forming the edge portion, respectively, in the vicinity of the edge portion so as to sandwich the edge portion; A method for aligning a virtual space with a real space, wherein the installation positions of the two markers, which are installed on either side of the edge, in a direction along the edge are such that the installation ranges of the markers in the direction along the edge at least partially overlap each other.

2. 2. The method for aligning a virtual space with a real space according to claim 1, wherein the step of placing the markers includes placing the markers near corners where three planes in the real space intersect.

3. 3. The method for aligning a virtual space with a real space according to claim 1, wherein the reference point is a corner where three planes in the real space intersect.

4. 4. The method for aligning a virtual space with a real space according to claim 1, further comprising: setting a virtual marker in a virtual model of the virtual space so that the position and orientation of the marker in the real space coincide with those of the marker in the real space, based on the information obtained in the measuring step and the information obtained in the calculating step.

5. 5. The method for aligning a virtual space with a real space according to claim 1, wherein in the step of placing the marker, the marker is placed so that the edge and one side of the rectangular marker are parallel to each other.

Images