Marker setting method

Abstract

To efficiently set a plurality of markers in aligning a virtual space with a real space in an AR device using the markers.SOLUTION: A marker setting method includes the steps of: setting a plurality of markers in a real space at predetermined intervals; storing, in a storage unit, virtual data including a plurality of virtual markers corresponding to the markers in the real space; measuring misalignment between the markers in image data obtained by capturing the markers in the real space through an imaging unit of an AR device and the virtual markers in the virtual data superimposed on the image data; and adjusting the intervals of the markers according to the misalignment.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a marker installation method. [Background technology]

[0002] Conventionally, it has been known to use AR devices that display AR (augmented reality), such as AR glasses, smartphones, and tablet terminals, to overlay virtual images (e.g., 3D-CAD data, etc.) on real-world scenery or photographed images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-115957 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned AR device, in order to align the position of the virtual image with the scenery of the real world, it is necessary to obtain information such as the position, orientation, and movement of the AR device in the real world. In particular, since it is often difficult to use GPS in civil engineering and construction fields, it is possible to simultaneously estimate the self-position and create an environmental map using a method called VSLAM. In addition, in order to align the position of the virtual image with the scenery of the real world, a method is conceivable in which a marker such as an AR marker is placed in the real space and the marker is imaged by the imaging unit of the AR glasses, thereby referencing the position and orientation of the AR marker.

[0005] For example, when aligning a virtual image with a real-world scene using multiple markers, no consideration has been given to how to efficiently place the markers.

[0006] Therefore, the present disclosure has been made in consideration of the above technical problems, and has an objective to provide a marker installation method that can efficiently install multiple markers when aligning a virtual space with a real space in an AR device using markers. [Means for solving the problem]

[0007] The marker installation method according to the present disclosure includes the steps of: installing a plurality of markers at predetermined intervals in a real space; storing, in a storage unit, virtual data including a plurality of virtual markers corresponding to the plurality of markers in the real space; measuring a deviation between the marker in image data obtained by capturing the marker in the real space with an imaging unit of an AR device and the virtual marker in virtual data displayed superimposed on the image data; and adjusting the intervals between the plurality of markers in accordance with the deviation. Effect of the Invention

[0008] According to the present disclosure, it is possible to provide a marker installation method that can efficiently install multiple markers when aligning a virtual space and a real space in an AR device using markers. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of an AR system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an AR device according to the present embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of a server according to the embodiment. [Figure 4] 1A to 1C are diagrams illustrating an example of installation of an AR marker according to the present embodiment. [Diagram 5] FIG. 2 is a flowchart illustrating a self-position estimation method for the AR device according to the present embodiment. [Figure 6]11A to 11C are diagrams illustrating other examples of installation of the AR marker according to the embodiment. [Figure 7] FIG. 4 is a flowchart illustrating a method for aligning a real space and a virtual space according to the embodiment. [Figure 8] 1 is a plan view showing an example of installation of a marker according to the present embodiment. FIG. [Figure 9] FIG. 4 is a flowchart illustrating a method for aligning a virtual space with a real space according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] Fig. 1 shows an example of an AR system according to an embodiment of the present invention. As shown in Fig. 1, the AR system of this example includes an AR device 10 and a server 20. The AR device 10 and the server 20 are connected via a network 30 and are capable of communicating with each other.

[0012] <ARデバイス> The AR device 10 in this example is AR glasses that are glasses-type information processing terminals. Note that the AR device 10 is not limited to AR glasses, and may be an information processing device such as a smartphone or a tablet terminal.

[0013] 2 is a block diagram illustrating an outline of the configuration of the AR device 10. As illustrated in the figure, the AR device 10 includes, for example, a control unit 11, a storage unit 12, a display unit 13, a communication unit 14, an imaging unit 15, and a sensor unit 16, which are connected to each other via a bus 17.

[0014] The control unit 11 transfers data between the various units and controls the entire AR device 10. For example, the control unit 11 is realized by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The control unit 11 can access the storage unit 12 and can store information in the storage unit 12 and read information from the storage unit 12.

[0015] The control unit 11 acquires, generates, outputs, etc., various types of data by executing programs stored in the storage unit 12. The control unit 11 acquires information acquired from other devices such as the imaging unit 15, the sensor unit 16, and the server 20, and can estimate the position and attitude (orientation) of the AR glasses based on the acquired information and the information of the storage unit 12, etc.

[0016] The storage unit 12 stores programs, codes, etc. for performing one or more steps, and the control unit 11 controls the operation of each unit according to the programs, etc. The storage unit 12 has a nonvolatile storage device which is a read-only storage area in which a system program is stored, and a volatile storage device which is a rewritable storage area used as a work area for arithmetic processing by the control unit 11. The nonvolatile storage device is realized by, for example, a ROM (Read Only Memory), a flash memory, a hard disk, etc., and the volatile storage device is realized by a RAM (Random Access Memory), a VRAM (Video Random Access Memory), etc.

[0017] The storage unit 12 may include, for example, a separable medium such as an SD card or a random access memory (RAM) or an external storage device. The storage unit 12 can store each piece of data acquired from the imaging unit 15 and the sensor unit 16. For example, image data such as still images and videos captured by a camera as the imaging unit 15 may be stored in a built-in memory or an external memory.

[0018] The display unit 13 includes a display device such as a liquid crystal display device. The display device can be, for example, a wave guide display. An image controlled by the control unit 11 is displayed on the display device. That is, the display unit 13 can display an image generated by the control unit 11. The display device also has a transparent portion that is partially or entirely transparent or semi-transparent, and can display a virtual image superimposed on a scene in real space while visually recognizing the scene through the transparent portion. The display unit 13 may include one or more lenses in addition to the display device.

[0019] The display unit 13 displays a virtual image (3D-CAD model, CG model, etc.) corresponding to the position (imaging position) and posture (imaging direction) of the AR glasses when an image of real space (for example, a marker described later) is captured (photographed) by superimposing it on the scenery of real space seen through the lens of the display device. Note that the display unit 13 may generate an AR image by superimposing a virtual image on an image captured by the imaging unit 15 using the control unit 11, etc., and display the AR image.

[0020] The communication unit 14 connects the AR device 10 to a network 30 including the Internet. The communication unit 14 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0021] The imaging unit 15 includes one or more imaging devices (cameras). The control unit 11 can send image data of still images or videos captured by the imaging devices to the storage unit 12, the server 20, etc. The imaging devices can include, for example, an RGB camera, a pair of left and right fisheye cameras for VSLAM, a ToF camera (Time-of-Flight Camera), etc. Such imaging devices can also function as the sensor unit 16. For example, a ToF camera (Time-of-Flight Camera) can measure the distance to an object by measuring the flight time of light, and is also called a ToF sensor.

[0022] The imaging unit 15 can capture continuous images for a predetermined time in response to a user's operation or based on preset instruction information. The imaging unit 15 can also capture non-continuous images at one point in time in response to a user's operation or based on preset instruction information. In other words, the imaging unit 15 can capture still images and moving images.

[0023] The sensor unit 16 can be, for example, a three-axis acceleration sensor, a three-axis gyro sensor, a direction sensor, a GPS, etc. Each sensor can transmit (input) to the control unit 11 information (detection signals) such as the position, attitude, moving speed, acceleration, etc. detected by each sensor.

[0024] The AR glasses may include an input unit that accepts input of requests (instructions) from a user or the like. The input unit may be, for example, a touch panel or a mechanical switch. The input unit may accept gaze input using eye tracking technology or voice input using a microphone or the like. The input unit may accept input from another information processing device such as a smartphone. The AR glasses may include an output unit such as a speaker that outputs voice or the like.

[0025] <Server> The server 20 is an information processing device used by a system administrator or the like when operating and managing various services, and may be, for example, a general-purpose computer such as a workstation or personal computer, or may be logically realized by cloud computing technology.

[0026] As shown in Fig. 3, the server 20 includes, for example, a control unit 21, a storage unit 22, an output unit 23, a communication unit 24, and an input unit 25, which are connected to one another via a bus 26. The server 20 is capable of communicating with the AR glasses via the communication unit 24. When the server 20 receives various request signals from the AR glasses or the like, the control unit 21 executes processing according to a program, and transmits the processing results (for example, generated images, sounds, etc.) to the AR glasses or the like as appropriate, or stores them in the storage unit 22. Note that a portion of the above program may be transmitted to the AR glasses and executed on the AR glasses.

[0027] The control unit 21 is responsible for transferring data between each part and controlling the entire server 20, and is realized by a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) executing programs stored in a designated memory (storage unit).

[0028] The storage unit 22 has a nonvolatile storage device which is a read-only storage area in which a system program is stored, and a volatile storage device which is a rewritable storage area used as a work area for arithmetic processing by the control unit 21. The nonvolatile storage device is realized by, for example, a ROM (Read Only Memory), a flash memory, or a hard disk, while the volatile storage device is realized by, for example, a RAM (Random Access Memory), a VRAM (Video Random Access Memory), or the like.

[0029] The storage unit 22 can store information related to virtual space, information related to virtual images, information related to markers, information related to each coordinate system (real space coordinate system, AR device coordinate system, marker coordinate system, virtual space coordinate system), and the like.

[0030] The storage unit 22 can store information received from an AR camera, etc. The storage unit 22 may also store information stored in the storage unit 12 of the AR camera.

[0031] The output unit 23 is composed of a speaker for outputting sound, a display for displaying images, and the like.

[0032] The communication unit 24 connects the server 20 to a network 30 including the Internet. The communication unit 24 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0033] The input unit 25 may be a keyboard, a mouse, a touch panel, or the like.

[0034] FIG. 4 is a diagram showing a method for improving the accuracy of self-location estimation of the AR device 10 by using AR glasses as the AR device 10 and markers 40 installed on the wall surface of a structure 50 at the site.

[0035] <marker> The marker 40 may be, for example, AprilTag, but is not limited thereto, and may be other AR markers (AR tags) such as ARTag, ARToolkit, ARToolkitPlus, RUNE-Tag, reacTIVison, etc. The marker 40 is, for example, a square as a whole. The marker 40 has a matrix structure in which a plurality of square pixels are arranged two-dimensionally, and each pixel is displayed in white or black. The shape of the marker 40 and the drawn figure are not limited thereto.

[0036] When AprilTag is used as the marker 40, for example, the 36h11 mark series is composed of 587 markers. By simply capturing an image of any marker 40 with the camera of the AR device 10, it is possible to obtain ID (identification information) unique to each marker 40. When capturing an image of any marker 40 with the camera of the AR device 10, the marker 40 may be captured from the front or from an oblique angle, and the identification information, position, and orientation information unique to each marker 40 can be obtained by image processing by the control unit. Images of each marker 40 and ID information corresponding to each image are associated with each other in advance and stored in the storage unit 22 (or storage unit 12).

[0037] Specifically, for example, when image data captured by the camera of the AR glasses is transmitted to the server 20, the control unit 21 of the server 20 refers to the data stored in the storage unit 22 and transmits ID information corresponding to the marker 40 to the AR glasses. This allows the control unit 11 of the AR glasses to obtain the ID corresponding to the marker 40. For example, information such as the position and orientation in real space can be associated with the ID information and stored in the storage unit 22. This allows the marker 40 to be accurately grasped not only by capturing an image of the marker 40 with the camera, but also by information such as the position and orientation of the marker in real space.

[0038] Such information about the marker 40 can be displayed on the display unit 13 of the AR glasses. The user can check information about the marker 40, such as the ID, position, and attitude, from the display unit 13 simply by capturing an image of the marker 40 with the camera of the AR glasses. Note that the process of acquiring the ID of the marker 40 based on the image of the marker 40 captured by the camera may be a series of processes performed by the AR camera alone (without communicating with the server 20). In this case, information about the marker 40 is stored in advance in the storage unit 12 of the AR camera.

[0039] <Coordinate system> 4, a coordinate system (AR device coordinate system) specific to the AR glasses is set for the AR glasses. Also, a coordinate system (marker coordinate system) specific to the marker 40 is set for the marker 40. Furthermore, a real space coordinate system (world coordinate system) is set for the real space, and a virtual space coordinate system (virtual space coordinate system) is set for the virtual model.

[0040] Each coordinate system is, for example, a three-dimensional Cartesian coordinate system, and has three orthogonal coordinate axes (x-axis, y-axis, z-axis) and an origin where the three coordinate axes intersect. Information on each coordinate system can be stored in storage units 12 and 22. That is, the origin position of each coordinate system, the direction of the coordinate axes, and the relative positions and attitudes (directions) of each coordinate system can be stored in storage units 12 and 22. Note that the position can be expressed by three-dimensional coordinates, and the attitude can be expressed by three-dimensional directions (the rotation angles around each of the x-axis, y-axis, and z-axis).

[0041] <Virtual data> The storage unit 22 can store information on a three-dimensional virtual space (virtual space) and information on a virtual model (three-dimensional or two-dimensional CG model) positioned at a predetermined position and posture in the virtual space. The virtual data has information on a virtual space coordinate system. It is preferable that the virtual space data corresponds to space data of a predetermined real space (for example, a construction site). The position, direction, and coordinate system in the virtual space are associated with the position, direction, and coordinate system in the real space. The origin position of the virtual space coordinate system can be made to coincide (or correspond) with a predetermined position in the real space. In addition, the three coordinate axes of the virtual space coordinate system can be made to coincide (or correspond) with the three coordinate axes set in the real space. In addition, the shape, position, and orientation of the virtual model in the virtual space correspond to, for example, the shape, position, and orientation of a building or the like provided at an actual construction site in the real space.

[0042] The virtual model can be created from, for example, 3D-CAD data of the object or BIM (Building Information Modeling) data. The method of creating the virtual image and the type of image data are not particularly limited. The storage unit 12 has position information of the virtual model with respect to the origin of the coordinate system of the virtual space and attitude information indicating the direction in which the virtual model faces in the virtual space coordinate system. The setting of the virtual space and the arrangement of the virtual model are performed in advance based on design data such as BIM data. The virtual model may be based on a plan view (for example, a 2D construction drawing or a detailed plan view). The generation of the virtual model may be performed by the server 20 or by another device. The AR glasses may acquire data related to the virtual model from the server 20 in advance, or may repeatedly acquire data related to the virtual model continuously (or intermittently) while the AR glasses are in use.

[0043] The origin and each coordinate axis of the AR device coordinate system are set at a predetermined position and orientation with respect to the AR glasses. The control unit 11 (or the control unit 21) can estimate the position and orientation of the AR glasses based on an image captured by the imaging unit 15 of the AR glasses, and can make the virtual data correspond to the real space coordinate system.

[0044] The AR glasses are devices that display a virtual image of an object superimposed on a real-space landscape (or an image of the real world) based on an image (video) of the real space captured by the imaging unit 15. The AR glasses are used as portable devices, such as worn on the head by users such as workers and supervisors at civil engineering, construction, railway, and traffic sites. Virtual images of components and buildings to be installed are displayed superimposed on the real-space landscape (or an image of the real world) at the intended installation position and orientation (posture) of the real space landscape at the site. The virtual images are generated in advance using 3D-CAD or the like, and stored in the storage units 12 and 22 of the AR device 10 or the server 20.

[0045] The AR glasses acquire information on the position and attitude (angle) of the AR glasses, and the control unit 11 acquires and refers to data from the storage unit 12 to generate image data corresponding to the position and attitude. The generated image data (virtual image data) is then displayed on the display unit 13. As a result, a virtual image corresponding to the position and attitude of the AR glasses is displayed on the AR glasses so as to be superimposed on the scenery of the real world. By checking the virtual image at the work site, the user can work while having an image of the work process or the final result, and the work can be reliably carried out while suppressing work errors. In addition, even when giving instructions from a remote location, instructions can be efficiently given by talking while looking at a three-dimensional virtual image.

[0046] The position information and attitude information (direction information) of the AR glasses can be acquired, for example, by a self-location estimation technique. Examples of the self-location estimation technique include techniques that utilize image information, such as Visual SLAM (Simultaneous Localization And Mapping) and VIO (Visual Inertial Odometry), and the AR glasses may be equipped with a camera, a sensor, or the like for implementing the above techniques. Such self-location estimation techniques are particularly effective in non-GPS environments. The AR glasses of this example can simultaneously perform self-location estimation and the creation of an environmental map (virtual space) from an image captured by the imaging unit 15, using so-called "Visual SLAM."

[0047] The AR glasses of this embodiment are glasses-type display devices worn on the head by a user, and may include one or more lenses (integral with or separate from the display unit 13), a frame (frame body) supporting the lenses, etc., a control unit 11, an imaging unit 15, a sensor unit 16, a display unit 13, a communication unit, a storage unit 12, an input unit, etc. Note that some components of the AR glasses (such as the control unit 11) may be separated as separate units and connected by wire or wirelessly, and the processing performed using the control unit 11 and the storage unit 12 may be executed by the server 20.

[0048] The position and attitude of the AR glasses are determined by a self-position estimation unit in the control unit 11. The AR glasses are provided with a communication unit for communication connection to an information processing device such as a server 20 via a network such as the Internet or directly. This allows various data to be transmitted and received between the AR glasses and other information processing devices provided with a communication unit. For example, image data captured by the AR glasses, image data generated by the control unit 11, sensor data acquired by the sensor unit 16, etc. can be transmitted to the server 20 and stored and processed on the server 20 side, and data such as images transmitted from the server 20 can be received.

[0049] Note that an image captured by the imaging section 15 of the AR device 10 may be transmitted to the server 20, and an image to be displayed on the AR device 10 may be generated in the control section 21 of the server 20.

[0050] The control unit 11 acquires information on the position (imaging position) and posture (imaging direction) of the imaging unit 15 (camera), generates an image of a virtual model corresponding to the position and posture of the imaging unit 15 when the image was captured (based on the information from the memory unit 12), and outputs it to the display unit 13.

[0051] The control unit 11 generates an image of the virtual model when the imaging unit 15 captures an image of the virtual model installed in the virtual space with a virtual camera, which is assumed to be in the virtual space, under the same conditions of position and orientation as when the image was captured by the imaging unit 15, and the display unit 13 displays the image so that it is superimposed on the scenery in the real space (or the image of the real space).

[0052] When displaying the virtual model on the display unit 13 of the AR glasses, not only the position and orientation of the camera but also other parameters such as the brightness of the real space may be taken into consideration. The virtual model may be made semi-transparent so that the overlapping portion (hidden portion) of the virtual model image can be visually recognized. The display unit 13 of the AR glasses sets and modifies the virtual image to be displayed in accordance with the position and orientation of the camera.

[0053] The control unit 11 can estimate the movement state (presence or absence of movement, movement direction, movement speed, etc.) of the imaging device of the AR glasses based on information from the imaging unit 15 or the sensor unit 16. The control unit 11 calculates the position and orientation of the camera in the virtual space (the position and orientation when the camera is in the virtual space) according to the movement state information. The display unit 13 overlays a virtual model on the image based on the corrected position and orientation. By continuously performing such processing, it is possible to display on the display unit 13 a virtual image that follows the movement of the AR glasses in accordance with the user's movement.

[0054] The display unit 13 generates an image to be displayed and outputs it to the display every time it inputs a captured image (continuously or intermittently) and every time it receives information indicating a moving state from a sensor unit or the like. That is, the display unit 13 displays a virtual model by superimposing it on a landscape or image in real space so as to follow the moving state of the camera. This function of the display unit 13 can be realized by a known AR method (for example, a tracking algorithm or a SLAM function of ARToolKit).

[0055] Hereinafter, a method for calibrating (correcting) the self-position and orientation using the marker 40 will be described with reference to FIGS.

[0056] First, as shown in Fig. 4, the marker 40 is positioned and installed in advance at a predetermined position on a construction site or the like (step S1). The position and attitude (direction) of the marker 40 in the real space coordinate system are stored in advance in at least one of the storage units (12, 22) of the AR glasses or the server 20. In the example of Fig. 4, the marker 40 is installed on a vertical wall surface of the structure 50, but it may be installed on a horizontal surface or the like. Also, the marker 40 is attached so that two sides are parallel to the upper edge (horizontal edge located at the upper end) and side edge (vertical edge located at the side end) of the wall surface of the structure 50, but this is not limited thereto.

[0057] The user adjusts the orientation of the camera of the AR glasses so that the marker 40 is located within the imaging range of the camera, and captures an image of the marker 40 with the camera (step S2).

[0058] The control unit (11, 21) calculates the three-dimensional position and orientation of the marker 40 on the AR device coordinate system (camera coordinate system) based on the image data of the captured marker 40. At that time, the position and orientation of the marker 40 are determined using a point constituting one of the corners of the square marker 40 and two sides (edges) perpendicular to the corner at that point as reference elements (reference point, reference line). The control unit (11, 21) calculates the three-dimensional relative positional relationship (coordinate transformation matrix (R, T)) between the AR glasses coordinate system and the marker coordinate system.

[0059] The control unit (11, 21) also calculates the position and orientation of the AR glasses on the marker coordinate system by performing inverse calculation using the coordinate transformation matrix (R, T) based on the position and orientation of the marker 40 on the AR device coordinate system. As a result, the control unit (11, 21) calculates the relative position and orientation of the AR glasses with respect to the marker 40 (step S3).

[0060] The control unit (11, 21) acquires the identification information, position, and orientation information of the marker 40 in the real space coordinate system from the storage unit (12, 22) based on the image data of the captured marker 40. In other words, the control unit (11, 21) acquires the three-dimensional relative positional relationship (coordinate transformation matrix (R0, T0)) between the marker coordinate system and the real space coordinate system from the storage unit (12, 22). The position and orientation of the marker 40 on the real space coordinate system and the relative three-dimensional positional relationship (position and orientation) between the real space coordinate system and the marker coordinate system are stored in advance in the storage unit (12, 22). The control unit (11, 21) solves a so-called PnP problem (Perspective n-Point) that determines an image of an object whose position and shape are known, and the position and orientation of the camera that captured the image, thereby making it possible to estimate the position coordinates (imaging position coordinates) and orientation of the camera and the AR glasses including the camera.

[0061] The control unit (11, 21) calculates the position and orientation of the AR glasses in the real space coordinate system based on information on the relative position and orientation of the AR glasses with respect to the marker 40 and the relative three-dimensional positional relationship between the real space coordinate system and the marker coordinate system (i.e., by performing inverse calculation using the coordinate transformation matrix (R0, T0)) (step S4).

[0062] Using this method, the AR glasses can estimate their own position and orientation in the real space coordinate system with high accuracy.

[0063] A method for aligning the virtual space with the real space will be described below with reference to Figures 6 and 7. More specifically, a method for initial setting when using an AR device on-site will be described.

[0064] The method for aligning the virtual space with the real space according to the present embodiment includes the steps of: placing a marker along a corner or edge in the real space; measuring the position and orientation of the marker with respect to the corner or edge of a structure; calculating the relative position and orientation of the AR device with respect to the marker based on image information obtained by capturing the marker with an imaging unit of the AR device; and aligning the coordinate system of the virtual space stored in advance in the storage unit with the coordinate system of the real space based on the information obtained in the measuring step and the information obtained in the calculating step. With this configuration, it is possible to perform alignment of the virtual space with the real space with high accuracy in any environment, i.e., in various environments.

[0065] Specifically, for example, when using the AR glasses for the first time on-site, the user performs initial setting (setup). First, an application stored in the AR glasses is started by turning on the power switch of the AR glasses, for example. Then, as initial setting, an initial alignment is performed between a virtual space coordinate system (virtual world) in which a three-dimensional virtual image is set and a real space coordinate system (real world). Information on the virtual space coordinate system in which the virtual image is set is stored in the storage units 12 and 22. The information on the virtual space coordinate system includes the virtual coordinate system, information on the virtual model, and information on the marker in the virtual space.

[0066] At the site, for example, near the entrance of a space where a building, structure, etc. will be installed, a marker 40 is installed near a position that is easy to refer to in a three-dimensional design drawing (such as 3D-CAD) (step S11). An example of a position that is easy to refer to is a corner (an outer corner or an inner corner) or an edge (e.g., a straight edge) of a building. When the control unit performs self-location estimation and mapping (map formation) using VSLAM or the like, an edge (edge) where two adjacent planes intersect, or a corner where three planes intersect can be set as reference point P. When installing the marker, it can be detachably fixed to the target object by attaching it using an adhesive means or the like.

[0067] 6, a corner (inner corner) where three mutually orthogonal surfaces (a horizontal floor surface and two vertical wall surfaces) in a building intersect is set as a reference point P. A first marker 40a is placed on the floor surface, and a second marker 40b is placed on the wall surface (surface of a side wall).

[0068] When installing the markers 40, it is preferable to install multiple markers 40 on different planes (wall surface, floor surface). This allows more information to be referred to than when only one marker 40 is used for alignment, thereby improving the accuracy of alignment. Note that the positions at which the markers 40 are installed are preferably, but not limited to, a vertical wall surface or a horizontal floor surface. In addition, from the viewpoint of improving the accuracy of alignment, it is preferable to install the markers 40 on planes that are perpendicular to each other. This allows the accuracy of alignment to be improved compared to when the markers are installed on planes that extend obliquely relative to each other.

[0069] It is also preferable to place markers 40 on each of a plurality of wall surfaces constituting the corner that is the reference point P. This allows a plurality of markers to be placed at different angles around the corner that is the reference point P, thereby further improving the accuracy of alignment.

[0070] It is also preferable to place multiple markers 40 (i.e., in relatively close positions) within an area that can be simultaneously imaged by the imaging unit 15 of the AR device 10. In the example of Fig. 6, a first marker 40a and a second marker 40b are placed on planes (floor surface and wall surface) that are perpendicular to each other. Note that, as in this example, it is preferable to place the two markers 40 so that they sandwich an edge (a linear edge where two planes intersect) from both sides.

[0071] The marker 40 is preferably installed along the corners and edges of the building. More specifically, the marker 40 is preferably installed so that two perpendicular sides of the wall surface constituting the corner of the inner wall surface of the building and two sides forming the corner of the square marker are parallel to each other. Alternatively, the marker 40 is preferably arranged so that one side of the square marker 40 is parallel to any edge of a structure such as a building. In this way, the marker 40 can be easily installed in an appropriate position and orientation, and the positioning of the virtual space with respect to the real space can also be easily performed. The marker may be installed in a predetermined position (latitude, longitude) and in a predetermined orientation (for example, one side of the marker is parallel to any of the directions of east, west, north, south, etc.) and positioned on the ground. The marker 40 can be arranged by offsetting it along a plane with respect to the reference corner or edge by a method such as marking.

[0072] Using a measuring device such as a laser positioning meter, data (R0, T0) indicating the relative deviation of each marker 40 from the reference point P and coordinate axis (azimuth axis) of the 3D-CAD design drawing (i.e., the relative position and orientation of the marker 40) is measured (step S12). Specifically, in the real space, the position and orientation of the marker 40 are measured (actually measured) using the corner (reference point P) and edge of the building as a reference. This makes it possible to calculate the relationship (position, orientation) between the real space coordinate system and each marker coordinate system. Information on the position and orientation of the marker 40 in the real space is stored in the storage unit 22 of the server 20. The server 20 transmits the information to the AR glasses. Note that data indicating the deviation (coordinate transformation matrix (R0, T0)) may be stored in the storage unit 12 of the AR glasses. The measuring device is not limited to a laser positioning meter, and may be any device that can measure the distance, orientation, etc. between two specific points in the real space. For example, it may be a spirit level or a telescope that measures angles and distances from a long distance. Also, for example, a measuring method using marking including an automatic marking device, a laser marking device, etc. may be used. The automatic marking device can measure the relative position of the device to a reference point to determine the current position, or draw a line on the floor at the position of the reference line. The measuring device may be, for example, a device that calculates the shape and dimensions of the floor, wall, and reference point from an image captured by a camera. Alternatively, the measuring device may be a device that obtains the dimensions of the floor using a laser distance measuring device.

[0073] Then, at the site, the image capturing unit 15 of the AR glasses captures the images of the installed markers 40 (40a, 40b) (step S13). The control unit of the AR glasses automatically detects the markers 40. The control unit calculates the relative position and orientation of the AR glasses with respect to each marker 40 (40a, 40b) (step S14). Then, using information on the relative position and orientation of the AR glasses with respect to each marker 40 (40a, 40b) and the data (R0, T0) indicating the above-mentioned deviation, the initial alignment of the virtual space with respect to the real space is realized (step S15). That is, the real space coordinate system and the virtual space coordinate system are associated. Preferably, the origin position and the directions of the three coordinate axes of the real space coordinate system are made to coincide with the origin position and the directions of the three coordinate axes of the virtual space coordinate system.

[0074] In addition, as in this example, it is preferable to place the markers 40 on two orthogonal planes, which can further improve the accuracy of alignment. Also, by placing the markers 40 on three orthogonal planes, it is possible to further improve the accuracy of alignment.

[0075] The technology disclosed herein is particularly effective in places with few feature points, such as tunnel construction, places with many similar landscapes, or in situations where the environment is constantly changing. Fields that have such problems include civil engineering, architecture, railways, and transportation.

[0076] Hereinafter, a method for calibrating (correcting) the self-position and attitude using a plurality of markers 40 will be described with reference to Figs. 8 and 9. Fig. 8 is a plan view showing a case where a plurality of markers are installed at a predetermined interval from the entrance inside a building in real space. The markers may be installed on the floor or wall, or may be installed at another position. In the example of Fig. 8, six markers are installed, but this is not limited to this. As described above, it is preferable to install each marker 40 along a corner or edge of the current space-time that serves as a reference point.

[0077] The marker installation method according to the present embodiment includes the steps of: installing a plurality of markers at a predetermined interval in a real space; storing virtual data including a plurality of virtual markers corresponding to the plurality of markers in the real space in a storage unit; measuring a deviation between the marker in image data obtained by capturing the marker in the real space with an imaging unit of an AR device and the virtual marker in virtual data displayed superimposed on the image data; and adjusting the interval between the plurality of markers according to the deviation. With this configuration, it is possible to efficiently install a plurality of markers when aligning the virtual space and the real space in the AR device using the markers.

[0078] 8 and 9, a plurality of markers 40 are placed at regular intervals in real space (step S21). For example, a first marker 40 for the above alignment is placed near an entrance to a space such as a building or structure. The plurality of markers 40 may be the same or different types of markers 40.

[0079] Next, the second marker 40 is placed according to the space so that the distance from the first marker 40 is a specific distance (for example, 10 m). Similarly, the third marker 40 is placed according to the above-mentioned specific distance from the second marker 40, and the fourth marker 40 is placed according to the specific distance from the third marker 40. In this way, the markers 40 are placed at equal intervals according to the space (situation) of the site. Note that the specific distance is not particularly limited and is a value that can be changed as appropriate.

[0080] Here, a virtual marker is set in the virtual space (step S22). The virtual marker may be set before or after step S21 of placing the marker in the real space. For example, in a virtual model of the virtual space stored in the storage unit, each marker 40 is drawn (3D modeled) and stored so as to match the position and posture of the marker in the real space. As a result, a virtual marker corresponding to the marker 40 in the real space is displayed as a virtual image displayed on the display unit 13 of the AR glasses. A virtual model including walls and floors corresponding to (having the same shape as) structures including walls and floors installed at the site in the real space is stored in advance in the storage unit as virtual data positioned in the virtual space.

[0081] Then, while checking the virtual image displayed on the AR glasses, the user moves from the entrance of the site in the real space to the space, capturing images of the first marker, the second marker, the third marker, and so on, with the imaging unit of the AR glasses. At this time, the position and orientation of each virtual marker (virtual marker displayed on the AR glasses) assumed to be displayed superimposed on each marker 40 in the real space is compared with the position and orientation of the marker 40 in the real space. The AR glasses continue to estimate their own position and orientation using VSLAM or the like, and display a virtual image corresponding to the estimation on the display unit. Then, it is confirmed whether or not the amount of deviation between the position and orientation of the marker in the real space and the position and orientation of the virtual marker exceeds a predetermined threshold (step S23). The threshold may be a value previously determined by the administrator and stored in the storage unit, or may be set by the user by inputting a numerical value via the input unit or selecting from a plurality of previously set options. The threshold for position deviation is indicated by a distance such as 5 mm, and the threshold for posture deviation is indicated by an angle such as 5°.

[0082] The measurement of the amount of deviation can be performed, for example, by the AR glasses or the control unit of the server. When the user captures the image of the marker with the imaging unit of the AR glasses, the control unit superimposes the image captured by the imaging unit of the AR glasses with a virtual image to be displayed on the display unit of the AR glasses, and measures the deviation between the position and attitude of the marker in the image captured in the real space and the position and attitude of the virtual image. More specifically, the marker coordinate system of the marker in the image captured in the real space can be compared with the virtual marker coordinate system of the virtual marker in the virtual space. The distance between the origin of the real marker coordinate system and the origin of the virtual marker coordinate system can be measured, or the coordinate axes of the real marker coordinate system and the coordinate axes of the virtual marker coordinate system can be compared to measure the difference (angle) between them. In addition, for example, reference points can be compared and measured. For example, the distance between specific corners of the rectangular marker 40 can be measured, or the distance between the sides and the relative angle can be measured to measure the deviation between the marker in the real image and the marker in the virtual image. The method of measuring the deviation is not limited to this.

[0083] Then, if the amount of deviation exceeds the threshold, the distance between the markers 40 is adjusted (step S24). For example, if the amount of deviation between the actual second marker 40 and the second virtual marker exceeds 10 mm, the distance between the markers 40 is shortened. Specifically, as a result of checking the amount of deviation with the distance between the markers 40 set to 10 m as described above, if the amount of deviation exceeds the threshold, the distance between the markers is shortened by 2 m to 8 m, for example, to a distance shorter than the initial distance of 10 m. In this example, the position of the second marker is first adjusted, and the second marker is moved (placed in a close position) so as to be closer to the first marker, for example. The extent to which the distance is adjusted to be shorter may be determined in advance and stored in the storage unit, or may be determined by the user. Alternatively, the adjustment distance may be determined according to the magnitude of the amount of deviation. In that case, if the amount of deviation is large, the adjustment distance may be increased, and conversely, if the amount of deviation is small, the adjustment distance may be reduced. In the above example, the adjustment distance is set to 2 m. If there is almost no amount of deviation, the distance between the markers may be increased. In this case, the extent to which the distance is increased may be determined in advance and stored in a storage unit, or may be determined by the user. Alternatively, the adjustment distance may be determined according to the magnitude of the amount of deviation.

[0084] After adjusting the position and orientation of the marker in real space, the position of the virtual marker in the virtual space is changed accordingly and stored in a storage unit. At this time, the position and orientation of the marker in the current space-time may be measured using a measuring device such as a laser positioning device. This allows modeling of the virtual marker with a more accurate position and orientation.

[0085] As described above, by checking the position and orientation deviation between the marker in real space and the virtual marker in the virtual image, the accuracy of the movement detection of the AR glasses can be checked. Then, it can be checked whether the distance between the markers 40 is appropriate (depending on the accuracy of the AR glasses). As a result, the markers 40 can be placed with an appropriate feel, and the efficiency of using the markers 40 can be improved. In other words, it is possible to prevent the wasteful use of many markers 40.

[0086] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0087] The device described in this specification may be realized as a single device, or may be realized by a plurality of devices (e.g., cloud servers) partly or entirely connected via a network. For example, the control unit 21 and the storage unit 22 of the server 20 may be realized by different servers connected to each other via a network.

[0088] The series of processes by the device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. It is possible to create a computer program for realizing each function of the AR device 10 and the server 20 according to this embodiment and to install it in a PC or the like. In addition, a computer-readable recording medium in which such a computer program is stored can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above computer program may be distributed, for example, via a network, without using a recording medium.

[0089] In addition, the processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may be employed, and some process steps may be omitted.

[0090] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0091] Note that the following configurations also fall within the technical scope of the present disclosure. (Item 1) placing a plurality of markers at predetermined intervals in a real space; storing, in a storage unit, virtual data including a plurality of virtual markers corresponding to the plurality of markers in the real space; measuring a deviation between the marker in image data obtained by capturing the marker in the real space with an imaging unit of an AR device and the virtual marker in virtual data displayed superimposed on the image data; and adjusting the spacing between the multiple markers in accordance with the deviation. (Item 2) 2. The marker installation method according to item 1, characterized in that in the step of measuring the deviation, a difference in at least one of a position and an orientation between a marker coordinate system of the marker in the captured image data and a virtual marker coordinate system of the virtual marker is measured. (Item 3) In the step of adjusting the interval, 3. The marker placement method according to item 1 or 2, further comprising reducing the spacing when the deviation is greater than a predetermined first threshold. (Item 4) In the step of adjusting the interval, The marker installation method according to any one of items 1 to 3, wherein the interval is increased when the deviation is smaller than a predetermined second threshold value. [Explanation of symbols]

[0092] 1. AR System 10. AR Devices 20 Servers 30 Network 40 Marker

Claims

1. storing, in a storage unit, virtual data including a plurality of virtual markers corresponding to a plurality of markers placed at a predetermined interval in a real space; Detecting a marker installed in a real space using an imaging unit of an AR device; and measuring a deviation between the marker in image data captured by an imaging unit of the AR device and the virtual marker in virtual data displayed superimposed on the image data.

2. 2. The marker installation method according to claim 1, wherein in the step of measuring the deviation, a difference in at least one of a position and an orientation between a marker coordinate system of the marker in the captured image data and a virtual marker coordinate system of the virtual marker is measured.

3. The AR device has a transparent portion configured to be transparent or semi-transparent, 3. The marker installation method according to claim 1, wherein while a scene in real space is visually recognized through the transparent portion, the image of the virtual data can be displayed superimposed on the scene.

4. 4. The marker installation method according to claim 1, further comprising: determining an adjustment amount for an installation interval between a plurality of markers in real space by comparing the measured deviation with a predetermined threshold value.

Images