Augmented reality display device, method, and program

Abstract

To provide an augmented reality display device or the like, which can contribute to alignment between a virtual image and a real image even in a site with many obstacles or a narrow site.SOLUTION: An augmented reality display device includes a storage unit storing a predetermined application and model data, and a control unit. The model data includes a plurality of virtual AR markers associated with position information in association with one of parts corresponding to corners of a plurality of walls. The control unit displays, when the predetermined application is started, a message so as to bring one end surface of the augmented reality display device into close contact with a corner of a wall applied with a nearest virtual AR marker, highlights the nearest virtual AR marker on the model data or image data, aligns the model data with the image data when the one end surface of the augmented reality display device is brought into close contact with the wall, and after alignment between the model dana and the image data, continuously performs alignment between the model data and the image data on the basis of positioned position information.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an augmented reality display device, method, and program. [Background technology]

[0002] In connection with the digital transformation (DX) of the manufacturing industry, there is a growing need at manufacturing production sites to manage on-site records (on-site personnel, dates and times, inspection results, test results, on-site photos, and notes) by mapping them on a digital twin in the form of virtual sticky notes. Digital twins here refer to 3D models of the site (for example, BIM (Building Information Modeling)) and 2D models (for example, site drawings, floor plans, and construction drawings). There is also a growing need to be able to refer to on-site records mapped on a digital twin even after production is complete, for maintenance and after-sales service purposes.

[0003] Therefore, technologies that meet these needs have begun to be provided. For example, Patent Documents 1 and 2 disclose systems that use AR (Augmented Reality) markers to align and display model data (such as drawing data) in real space. However, at sites (such as manufacturing sites and construction sites) where many obstacles such as parts and materials exist, it is often difficult to install (attach, place, etc.) AR markers. Furthermore, at sites that require the installation of multiple AR markers, such as large sites or sites with poor visibility, a series of tasks such as printing multiple AR markers and installing them in specified positions is required, which is time-consuming. Furthermore, once a project is completed, such as at a construction site, AR markers are often removed, and after removal of the AR markers, they must be reinstalled to align the virtual space with the real space.

[0004] Therefore, technologies have been provided that enable model data to be aligned with real space without using AR markers. For example, Patent Document 3 discloses a system that aligns a virtual image with a captured image by comparing feature points in the captured image with feature points in the virtual image and performing coordinate transformation to display the virtual image. Furthermore, Patent Document 4 discloses a system that aligns a live view image with a virtual space by matching an object in the live view image with a mark image registered in a virtual space (such as a map) and displays the live view image with the virtual space. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2018-163466 [Patent Document 2] Patent Publication No. 2020-194214 [Patent Document 3] Patent Publication No. 2014-002645 [Patent Document 4] Patent Publication No. 2020-098568 Summary of the Invention [Problem to be solved by the invention]

[0006] The following analysis is provided by the present inventors.

[0007] However, even with the systems described in Patent Documents 3 and 4, which do not use AR markers, at work sites where there are many obstacles such as parts and materials, feature points (objects) often become hidden in real images (captured images, live view video, etc.) by the obstacles, which may make it impossible to align the real image with the virtual image (virtual space, map, etc.).Furthermore, with the systems described in Patent Documents 3 and 4, at narrow work sites (for example, rooms, corridors, structures within a few meters), feature points (objects) often do not appear in the captured images, which often makes it impossible to align the real image with the virtual image.

[0008] The main objective of the present invention is to provide an augmented reality display device, method, and program that can contribute to aligning a virtual image and a real image even in a location with many obstacles or a narrow space. [Means for solving the problem]

[0009] The augmented reality display device according to a first viewpoint is an augmented reality display device that displays model data aligned with image data of a photographed object, and includes a display unit configured to display information, an imaging unit configured to generate image data by photographing the object, a positioning unit configured to detect position information of the augmented reality display device, a memory unit configured to store a predetermined application and model data, and a control unit configured to control the display unit, the imaging unit, the positioning unit, and the memory unit and to perform predetermined information processing, wherein the model data is provided with a plurality of virtual AR markers associated with position information that aligns with any of positions corresponding to corners of a plurality of walls, and the control unit The control unit is configured to perform the following processes when the specified app is launched: aligning the augmented reality display device with the corner of the wall to which the nearest virtual AR marker out of the plurality of virtual AR markers is attached and displaying a message on the display unit to instruct the user to press one end face of the augmented reality display device against the wall; highlighting the nearest virtual AR marker on the model data or the image data on the display unit; aligning the model data with the image data when the one end face of the augmented reality display device is pressed against the wall; and, after aligning the model data with the image data, continuously aligning the model data with the image data based on the position information detected by the positioning unit.

[0010] An augmented reality display method according to a second aspect is an augmented reality display method that uses hardware resources to align model data with image data of a photographed object and display the aligned model data, wherein the model data is assigned with a plurality of virtual AR markers, each of which has associated position information corresponding to one of a plurality of wall corners; and when a predetermined application is launched, the method includes the steps of: aligning the hardware resource with the corner of a wall to which the nearest virtual AR marker of the plurality of virtual AR markers is assigned, and displaying a message to instruct the user to press one end face of the hardware resource against the wall; highlighting the nearest virtual AR marker on the model data or the image data; aligning the model data with the image data when pressing the one end face of the hardware resource against the wall; and, after aligning the model data with the image data, continuously aligning the model data with the image data based on position information detected by positioning.

[0011] A program according to a third aspect of the present invention is a program that causes a hardware resource to execute a process of aligning model data with image data of a photographed object and displaying the model data, wherein the model data is assigned with a plurality of virtual AR markers, each of which has associated position information corresponding to one of a plurality of wall corners. When a predetermined application is launched, the program causes the hardware resource to execute the following processes: aligning the hardware resource with the corner of a wall to which the nearest virtual AR marker of the plurality of virtual AR marker is assigned and displaying a message to press one end face of the hardware resource against the wall; highlighting the nearest virtual AR marker on the model data or the image data; aligning the model data with the image data when pressing the one end face of the hardware resource against the wall; and, after aligning the model data with the image data, continuously aligning the model data with the image data based on position information detected by positioning.

[0012] The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present disclosure can also be embodied as a computer program product. The program is input to a computer device from an input device or an external device via a communication interface, stored in a storage device, and drives a processor according to predetermined steps or processes. The processing results, including intermediate states as needed, can be displayed at each stage on a display device, or the computer device can communicate with the outside world via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and, if necessary, a display device, all of which can be connected to each other via a bus. [Effects of the Invention]

[0013] The first to third aspects can contribute to aligning a virtual image with a real image even in a site where many obstacles exist or a narrow site. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram schematically illustrating a configuration of an augmented reality display device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram schematically illustrating an example of model data displayed on the augmented reality display device according to the first embodiment. [Figure 3] 10A and 10B are conceptual diagrams schematically illustrating an operation when the top surface of the augmented reality display device according to the first embodiment is brought into close contact with a corner of a wall to which a virtual AR marker is attached. [Figure 4] 10A and 10B are conceptual diagrams illustrating an example of offset correction from the position of the imaging unit of the augmented reality display device according to the first embodiment to the edge of a wall that serves as the reference point of a right-corner type virtual AR marker. [Figure 5]10A and 10B are conceptual diagrams illustrating an example of offset correction from the position of the imaging unit of the augmented reality display device according to the first embodiment to the edge of a wall that serves as the reference point of a left-corner type virtual AR marker. [Figure 6] 3 is an image diagram of the augmented reality display device according to the first embodiment moving between virtual AR markers. FIG. [Figure 7] FIG. 2 is an image diagram schematically illustrating an example of a display image when the augmented reality display device according to the first embodiment automatically detects the nearest virtual AR marker. [Figure 8] 4 is a flowchart schematically showing the operation of a control unit in the augmented reality display device according to the first embodiment. [Figure 9] FIG. 10 is a block diagram schematically illustrating the configuration of an augmented reality display device according to a second embodiment. [Figure 10] FIG. 10 is an image diagram that schematically shows a flow from registration of an AR image to viewing of a VR image regarding anchor information of the augmented reality display device according to the second embodiment. [Figure 11] FIG. 10 is a block diagram schematically illustrating the configuration of an augmented reality display device according to a third embodiment. [Figure 12] FIG. 2 is a block diagram illustrating a configuration of hardware resources. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments will be described with reference to the drawings. Note that, where reference numerals are used in this application, they are intended solely to facilitate understanding and are not intended to limit the present invention to the illustrated embodiments. Furthermore, the following embodiments are merely illustrative and do not limit the present invention. Furthermore, connecting lines between blocks in the drawings, etc., referred to in the following description, include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. Furthermore, although not explicitly shown, input and output ports exist at the input and output ends of each connecting line in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, etc., shown in this disclosure. The same applies to input and output interfaces. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. The computer device is configured to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless.

[0016] [Embodiment 1] An augmented reality display device according to a first embodiment will be described with reference to the drawings. FIG. 1 is a block diagram schematically illustrating the configuration of the augmented reality display device according to the first embodiment. FIG. 2 is a block diagram schematically illustrating an example of model data displayed on the augmented reality display device according to the first embodiment. FIG. 3 is an image diagram schematically illustrating an operation of bringing the upper end surface of the augmented reality display device according to the first embodiment into close contact with a corner of a wall to which a virtual AR marker is attached. FIG. 4 is an image diagram schematically illustrating an example of offset correction from the position of the imaging unit of the augmented reality display device according to the first embodiment to the edge of the wall that serves as the reference point of a right-corner type virtual AR marker. FIG. 5 is an image diagram schematically illustrating an example of offset correction from the position of the imaging unit of the augmented reality display device according to the first embodiment to the edge of the wall that serves as the reference point of a left-corner type virtual AR marker. FIG. 6 is an image diagram illustrating the augmented reality display device according to the first embodiment moving between virtual AR markers. FIG. 7 is an image diagram schematically illustrating an example of a display image when the augmented reality display device according to the first embodiment automatically detects the nearest virtual AR marker.

[0017] The augmented reality display device 10 is a device that displays an augmented reality image (see FIG. 1). The augmented reality display device 10 is a device (computer device) equipped with functional units (e.g., a processor, a storage device, an input device, a communication interface, and a display device) that constitute a computer and that have a positioning (position, direction, etc.) function and an imaging function. For example, a smartphone, a tablet terminal, a personal computer, a glasses-type device, etc. can be used. The augmented reality display device 10 has a function of aligning and displaying a model (2 in FIG. 2) in model data 15b with a captured object 1 (e.g., a structure, wall, floor, etc.) in captured image data. By executing a predetermined program, the augmented reality display device 10 is realized as a configuration equipped with a display unit 11, an imaging unit 12, a positioning unit 13, an input unit 14, a storage unit 15, and a control unit 16.

[0018] The display unit 11 is a functional unit that displays information (see FIG. 1). The display unit 11 displays information under the control of the control unit 16. For example, the display unit 11 may be a display device such as a liquid crystal display or an organic EL (ElectroLuminescence) display, or a glasses-type display that is communicably connected to the augmented reality display device 10.

[0019] The imaging unit 12 is a functional unit that captures an image of the object 1 (see FIG. 1). The imaging unit 12 generates image data by capturing an image of the object 1. The imaging unit 12 captures an image of the object 1 under the control of the control unit 16. The imaging unit 12 can use, for example, an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and may further include a 3D sensor for feature detection (e.g., a ToF (Time of Flight) camera, a stereo camera, a 3D-LIDAR (Laser Imaging Detection And Ranging), a depth sensor, a ranging sensor, a distance camera, etc.). The imaging unit 12 may have a zoom lens that can capture images at a telephoto or wide angle. The imaging unit 12 may be a monocular or compound eye.

[0020] The positioning unit 13 is a functional unit that detects information (e.g., longitude, latitude, direction, etc.) related to the current position of the augmented reality display device 10 (see FIG. 1). The positioning unit 13 determines the current position of the augmented reality display device 10 under the control of the control unit 16. The positioning unit 13 may use a function that can continuously perform positioning by tracking the movement (amount of movement, change in acceleration, etc.) of the augmented reality display device 10 from a state where the augmented reality display device 10 has been accurately positioned, such as SLAM (Simultaneous Localization and Mapping) or IMU (Inertial Measurement Unit). The positioning unit 13 may also detect angle information related to the direction in which the optical axis of the imaging unit 12 is pointing and the angle of elevation using a direction sensor, a geomagnetic sensor, an acceleration sensor, etc. In addition, when performing provisional positioning before performing accurate positioning, the positioning unit 13 may additionally use, for example, GPS (Global Positioning System), radio wave intensity from multiple wireless base stations with pre-set position information, UWB (Ultra Wide Band), Bluetooth (registered trademark) 5 AoA (Angle of Arrival) (Quuppa), BLE (Bluetooth (registered trademark) Low Energy) beacon, GNSS (Global Navigation Satellite System), etc.

[0021] The input unit 14 is a functional unit that inputs information (see FIG. 1). The input unit 14 inputs information input by a user operation under the control of the control unit 16. The input unit 14 may be, for example, a touch panel, a keyboard, a mouse, a non-contact UI (User Interface), a gesture sensor, a microphone, a headset, or the like.

[0022] The storage unit 15 is a functional unit that stores various types of information (data, programs, etc.) (see FIG. 1). The storage unit 15 may be, for example, a storage device such as a random access memory (RAM), a solid state drive (SSD), or a hard disk drive (HDD). The storage unit 15 writes and reads information under the control of the control unit 16. The storage unit 15 stores an AR application 15a and model data 15b.

[0023] Here, the AR application 15a is application software that causes the control unit 16 to execute a process of aligning and displaying a model (2 in FIG. 2) in the model data 15b with the object 1 to be photographed in the image data photographed by the imaging unit 12. Details of the process performed by the AR application 15a will be described later.

[0024] The model data 15b is data related to the structure of a model (2 in FIG. 2) that virtually represents the object 1 to be photographed (e.g., a building). Examples of the model data 15b include two-dimensional data, three-dimensional data, drawing data, and a virtual space model. Virtual AR markers 3 (3a to 3c in FIG. 2) are attached to multiple locations on the model 2 (see FIG. 2). The virtual AR markers 3 are attached to locations corresponding to corners of multiple walls 7 on the model 2. The virtual AR markers 3 serve as landmarks for bringing the top end surface (one end surface) of the augmented reality display device 10 into close contact with the corners of the wall 7. The virtual AR marker 3 has a reference point 4 at a position corresponding to an edge 7a (corner) of the wall 7 (see FIG. 3). The reference point 4 serves as the reference position of the virtual AR marker 3, and is associated with position information (e.g., coordinates, longitude, latitude, direction, height, etc.). When the model data 15b is three-dimensional data, the height is also included as part of the position information. There are two types of virtual AR markers 3: one located at the right corner of the wall 7 (3b in FIG. 2) and one located at the left corner of the wall 7 (3a and 3c in FIG. 2). As a setting position for the virtual AR marker 3, for example, a virtual AR marker 3a may be added as a starting point to a corner of the wall 7 closest to the entrance in the drawing data as the model data 15b, and another virtual AR marker 3b may be added to a corner of the wall 7 in an area that is in the blind spot of the entrance. Taking into consideration a site with many obstacles, the virtual AR markers 3b and 3c may be added to nearby positions that are not in each other's blind spots.

[0025] The control unit 16 is a functional unit that controls the display unit 11, the imaging unit 12, the positioning unit 13, the input unit 14, and the storage unit 15 (see FIG. 1). The control unit 16 may be implemented by a processor such as a CPU (Central Processing Unit), an MPU (Micro Processor Unit), or a GPU (Graphics Processing Unit). The control unit 16 executes a predetermined program stored in the storage unit 15, thereby performing predetermined processing described in the program.

[0026] When the AR application 15a is started, the control unit 16 displays the model data 15b (which may be image data captured by the imaging unit 12) on the display unit 11. The control unit 16 detects information (position, direction, attitude, etc.) related to the current position of the augmented reality display device 10 using the positioning unit 13 (e.g., GPS), and displays a tentative position (corresponding to the device current position 5 in FIG. 2 ) and direction (the direction in which the top surface faces from the device center; corresponding to the device orientation 6 in FIG. 2 ) of the augmented reality display device 10 on the model data 15b on the display unit 11 based on the detected information and the position information of the virtual AR marker 3 assigned in the model data 15b (in the case of image data, the image data is temporarily aligned with the model data 15b before being displayed). The control unit 16 causes the display unit 11 to highlight the virtual AR marker 3 closest to the augmented reality display device 10 on the model data 15b (for example, by displaying it in a conspicuous color, enlarging it, flashing it, etc.; in the case of image data, the virtual AR marker is highlighted at the corner of the actual wall), and displays a message to align the right or left end face and top end face of the augmented reality display device 10 with the corner of the wall 7 to which the virtual AR marker 3 is attached and press it tightly against it.

[0027] The message display indicates that when the augmented reality display device 10 is placed in close contact with the wall, the display unit 11 faces upward (toward the ceiling) and the augmented reality display device 10 is positioned horizontally. At this time, the imaging unit 12 (corresponding to the rear camera) of the augmented reality display device 10 faces downward (toward the ground). If the virtual AR marker 3 is a right-corner type, the right end face (the other end face perpendicular to the top end face) of the augmented reality display device 10 is aligned with the right corner of the wall 7, and the top end face (one end face) of the augmented reality display device 10 is placed in close contact with the wall. If the virtual AR marker 3 is a left-corner type, the left end face (the other end face perpendicular to the top end face) of the augmented reality display device 10 is aligned with the left corner of the wall 7, and the top end face of the augmented reality display device 10 is placed in close contact with the wall.

[0028] At the time of close contact, the control unit 16 aligns the model 2 in the model data 15b with the photographed object 1 in the photographed image data, either automatically or in response to a user operating (e.g., tapping) the input unit 14. When aligning, not only the planar position but also the direction is aligned, and the distance (height) between the imaging unit 12 of the augmented reality display device 10 and the ground (floor) is measured to align the position in the vertical direction as well.

[0029] The accuracy of the planar alignment between the model 2 in the model data 15b and the captured object 1 in the image data can be improved by offsetting the lens position of the image capture unit 12 to the position of the reference point 4 of the virtual AR marker 3 in the model data 15b. However, the lens position differs depending on the model of the augmented reality display device 10. The control unit 16 performs offset correction for each model of the augmented reality display device 10 (see FIGS. 3 and 4) to accurately align the model 2 in the model data 15b with the captured object 1 in the image data. The offset correction may involve inputting the actual lens position values ​​(distances from the top, right, and left end faces to the lens center (e.g., Z1, X1, and X2 in FIGS. 3 and 4), or the distance from the lens surface to the image sensor), or using values ​​preset in the AR app 15a. When the image capture unit 12 is a compound lens, the offset correction can be performed using the center of gravity of the center positions of the multiple lenses as the reference position of the image capture unit 12. Alternatively, the offset correction may be performed according to the center positions of the lenses used.

[0030] Furthermore, when measuring the distance (height) between the imaging unit 12 of the augmented reality display device 10 and the ground (floor), the control unit 16 may use a spirit level function to measure the vertical distance from the imaging unit 12 of the augmented reality display device 10 to the ground when the surface of the display unit 11 of the augmented reality display device 10 is level, thereby aligning the model data 15b with the image data in the height direction. Furthermore, the control unit 16 may measure the orientation of the imaging unit 12 of the augmented reality display device 10 and the distance from the imaging unit 12 to the ground, and calculate the vertical distance from the imaging unit 12 of the augmented reality display device 10 to the ground based on the measured orientation of the imaging unit 12 and the distance from the imaging unit 12 to the ground, thereby aligning the model data 15b with the image data in the height direction. The distance (height) can be calculated based on, for example, an AR basic positioning function (distance measurement using point cloud data generated from image feature points), triangulation, measurement values ​​from a depth sensor, a LiDAR (light detection and ranging) sensor, etc. Note that if the model data 15b is a two-dimensional model, the two-dimensional model does not have a height position designation, but when aligning the two-dimensional model with the floor surface, the height between the imaging unit 12 of the augmented reality display device 10 and the floor is measured, and the two-dimensional model is aligned with the floor surface in the height direction.

[0031] Once the control unit 16 aligns the model data 15b with the captured image data using the virtual AR marker 3, even if the user of the augmented reality display device 10 moves, the control unit 16 continuously aligns the model data 15b with the captured image data while grasping the position of the augmented reality display device 10 using a predetermined positioning function (AR basic positioning function) in the positioning unit 13, such as SLAM (Simultaneous Localization and Mapping) or IMU (Inertial Measurement Unit). For example, as shown in FIG. 6, when the augmented reality display device 10 aligns the model data 15b with the captured image data using the virtual AR marker 3d at position (x1, z1), the control unit 16 continuously aligns the model data 15b with the captured image data from position (x2, z2) to position (x3, z3) while grasping the position of the augmented reality display device 10 using SLAM or IMU. However, as the user moves, errors in position and orientation gradually accumulate and eventually become non-negligible. When the error becomes too large to be ignored, the misalignment between the virtual AR marker 3 and the corner of the wall 7 at the site becomes noticeable (for example, it appears to be misaligned by about 10 cm). Because the error tends to increase as the movement distance of the augmented reality display device 10 increases, the control unit 16 may measure the movement distance (or the elapsed time) since the start of alignment using SLAM or IMU, and infer that the error has become too large to be ignored when the measured movement distance exceeds a preset threshold. When the error becomes too large to be ignored, the control unit 16 displays a message on the display unit 11, instructing the user to align the right or left end face or top end face of the augmented reality display device 10 with the corner of the wall 7 to which the nearest virtual AR marker 3 is attached. For example, as shown in FIG. 6, a message may be displayed when the error becomes too large to be ignored when the augmented reality display device 10 is at position (x2, z2). Furthermore, when the corner of the wall 7 to which the nearest virtual AR marker 3 is attached is included in the image data, the control unit 16 highlights the virtual AR marker 3 (see 3e in FIG. 7). This allows the user to bring the augmented reality display device 10 into close contact with the corner of the wall 7 to which the nearest virtual AR marker 3 is attached, if the user determines that there is a large deviation in position or orientation.During close contact, the control unit 16 aligns the model 2 in the model data 15b with the photographed object 1 in the photographed image data, either automatically or by the user operating (e.g., tapping) the input unit 14, as described above. For example, as shown in FIG. 6, the augmented reality display device 10 can be brought into close contact with the corner of the wall 7 to which the virtual AR marker 3e is attached at position (x3, z3) to align the model data 15b with the photographed data. Thereafter, the control unit 16 continuously aligns the model data 15b with the photographed data while grasping the position of the augmented reality display device 10 using a predetermined positioning function (e.g., SLAM or IMU) in the positioning unit 13.

[0032] Next, the operation of the control unit in the augmented reality display device according to embodiment 1 will be described with reference to the drawings. Fig. 8 is a flowchart schematically showing the operation of the control unit in the augmented reality display device according to embodiment 1. Please refer to Fig. 1 for the configuration of the augmented reality display device.

[0033] First, when the AR application 15a is started, the control unit 16 of the augmented reality display device 10 causes the display unit 11 to display the model data 15b (step A1).

[0034] Next, the control unit 16 detects information related to the current position of the augmented reality display device 10 using the positioning unit 13 (e.g., GPS), and based on the detected information and the position information of the virtual AR marker 3 assigned in the model data 15b, displays the virtual position (corresponding to the device current position 5 in Figure 2) and direction (the direction in which the top surface is facing from the center of the device; corresponding to the device orientation 6 in Figure 2) of the augmented reality display device 10 on the model data 15b on the display unit 11 (step A2).

[0035] Next, the control unit 16 causes the display unit 11 to highlight (for example, display in a conspicuous color or flash) the virtual AR marker 3 closest to the augmented reality display device 10 on the model data 15b, and displays a message instructing the user to align the right or left end surface and top end surface of the augmented reality display device 10 with the corner of the wall 7 to which the virtual AR marker 3 is attached and press it against the corner (step A3).

[0036] After step A3 or step A8, the control unit 16 aligns the right or left end face of the augmented reality display device 10 with the corner of the wall 7 to which the user has added the virtual AR marker 3, and brings the upper end face into close contact with the wall surface, and either when the user operates (e.g., taps) the input unit 14 or automatically, aligns and displays the model 2 in the model data 15b with the object 1 in the captured image data (step A4). Note that during alignment, vertical alignment and offset correction related to the lens position are performed.

[0037] After step A4, or if the amount of movement is less than the threshold (NO in step A7), the control unit 16 continuously aligns the model data 15b with the captured data and displays them (step A5) while grasping the position of the augmented reality display device 10 using the SLAM or IMU in the positioning unit 13. At this time, the control unit 16 measures the amount of movement after starting alignment using the SLAM or IMU.

[0038] Next, the control unit 16 determines whether or not the user has ended the AR application by operating the input unit 14 (step A6). If the AR application has been ended (YES in step A6), the process ends.

[0039] If the AR application has not been terminated (NO in step A6), the control unit 16 determines whether the measured movement amount is equal to or greater than a preset threshold (step A7). If the movement amount is less than the threshold (NO in step A7), the process proceeds to step A5.

[0040] If the amount of movement is equal to or greater than the threshold (YES in step A7), the control unit 16 causes the display unit 11 to display a message instructing the user to align the right end surface, left end surface, or top end surface of the augmented reality display device 10 with the corner of the wall 7 to which the nearest virtual AR marker 3 is attached, and highlights the virtual AR marker 3 when the corner of the wall 7 to which the nearest virtual AR marker 3 is attached is included in the image data (step A8). After step A8, the process proceeds to step A4.

[0041] According to embodiment 1, the augmented reality display device 10 is placed in close contact with the corner of the wall 7 corresponding to the virtual AR marker 3 assigned to the model data 15b, thereby aligning the model data 15b with the image data. This contributes to aligning the model data 15b (virtual image) with the image data (real image) even in a location where many obstacles exist or where the location is narrow.

[0042] Furthermore, according to embodiment 1, when aligning the model data 15b (virtual image) and the image data (real image), offset correction is performed taking into account the lens position of the imaging unit 12 of the augmented reality display device 10 relative to the edge 7a of the wall 7, which serves as the reference point, thereby improving the accuracy of alignment.

[0043] Furthermore, according to embodiment 1, in situations where accurate alignment between the model data 15b (virtual image) and the image data (real image) is required, the virtual AR marker 3 is highlighted and a message is displayed encouraging the user to place the augmented reality display device 10 close to the corner of the wall 7 corresponding to the virtual AR marker 3, thereby enabling accurate alignment to be performed smoothly.

[0044] Furthermore, according to embodiment 1, a provisional alignment before accurate alignment between the model data 15b (virtual image) and the image data (real image) is performed using UWB, Bluetooth (registered trademark) 5 AoA, BLE beacon, GNSS, etc., which have low alignment accuracy, and accurate alignment is performed by closely adhering the augmented reality display device 10 to the corner of the wall 7 corresponding to the virtual AR marker 3, thereby making it possible to improve the efficiency of alignment.

[0045] [Embodiment 2] A virtual reality display device according to embodiment 2 will be described with reference to the drawings. Fig. 9 is a block diagram schematically showing the configuration of the augmented reality display device according to embodiment 2. Fig. 10 is an image diagram schematically showing the flow from registration of an AR image related to anchor information of the augmented reality display device according to embodiment 2 to viewing of a VR image.

[0046] Embodiment 2 is an application of embodiment 1, in which the augmented reality display device 10 is placed in close contact with the corner of the wall 7 corresponding to the virtual AR marker 3 assigned to the model data 15b to align the model data 15b with the image data, and while the position of the augmented reality display device 10 is grasped using SLAM or IMU and the model data 15b is continuously aligned with the captured data, a specified anchor 9 is placed at an arbitrary position relative to the virtual AR marker in the AR image, and anchor information containing arbitrary information related to the position (position of the anchor 9, notes, photo, registered person's name, registration date and time, etc.) is created and can be displayed in the VR image (see Figures 9 and 10).

[0047] The storage unit 15 stores VR data 15c and a database 15d. The VR data 15c and the database 15d may be stored in an external server (not shown). The VR data 15c is virtual data corresponding to image data of the photographed object 1, and may be three-dimensional data (BIM, etc.) created by a designer or data generated by photographing using a three-dimensional sensor such as a depth sensor, LiDAR, or depth sensor. The VR data 15c is positionally associated with the model data 15b (including virtual AR markers). The database 15d stores anchor information.

[0048] While the control unit 16 is continuously aligning the model data 15b with the image data based on the position information detected by the positioning unit 13 (i.e., while the AR image is being displayed), the user operates the input unit 14 to specify an arbitrary position relative to the virtual AR marker (3a to 3c in FIG. 2) that served as the basis for alignment, and inputs arbitrary information to create anchor information, which is then stored in the database 15d. When the VR data 15c is displayed on the display unit 11, the control unit 16 reads the anchor information from the database 15d and displays (superimposed or skeletonized) the anchor information (the anchor only, or information associated with the anchor) at a position in the VR data 15c that corresponds to the position specified in the anchor information. This allows the user to view the anchor information at the specified position while referring to the VR data 15c of the subject 1 to be photographed.

[0049] The other configurations and operations are the same as those in the first embodiment.

[0050] According to the second embodiment, as in the first embodiment, it is possible to contribute to aligning the model data 15b (virtual image) with the image data (real image) even in a site where many obstacles exist or a narrow site, and since the augmented reality display device 10 is placed in close contact with the corner of the wall 7 corresponding to the virtual AR marker 3 assigned to the model data 15b to accurately align the model data 15b with the image data, it is possible to reduce the effort required for storing and reproducing the anchor information related to the anchor 9, and it is possible to achieve stable operation even in areas where the same structure is continuous, and it is possible to reproduce the anchor information over a wide range.

[0051] [Embodiment 3] A virtual reality display device according to embodiment 3 will be described with reference to the drawings. Fig. 11 is a block diagram showing a schematic configuration of the augmented reality display device according to embodiment 3.

[0052] The augmented reality display device 10 is a device that aligns model data 15b with image data of a photographed object 1 and displays the image. The augmented reality display device 10 includes a display unit 11, an imaging unit 12, a positioning unit 13, a storage unit 15, and a control unit 16. The display unit 11 is configured to display information. The imaging unit 12 is configured to generate image data by photographing the object 1. The positioning unit 13 is configured to detect position information of the augmented reality display device 10. The storage unit 15 is configured to store a predetermined application 15a and model data 15b. The control unit 16 is configured to control the display unit 11, the imaging unit 12, the positioning unit 13, and the storage unit 15, and to perform predetermined information processing.

[0053] The model data 15b is provided with a plurality of virtual AR markers (e.g., 3a to 3c in FIG. 2) associated with position information that corresponds to one of the locations corresponding to the corners of a plurality of walls (e.g., 7 in FIG. 2).

[0054] When a predetermined application 15a is launched, the control unit 16 performs a process of aligning the augmented reality display device 10 with a corner of a wall to which the nearest virtual AR marker out of multiple virtual AR markers is attached, and displaying a message on the display unit 11 to instruct the user to bring one end surface of the augmented reality display device 10 into close contact with the wall. The control unit 16 performs a process of highlighting the nearest virtual AR marker on the model data 15b or image data on the display unit 11. The control unit 16 performs a process of aligning the model data 15b with the image data when one end surface of the augmented reality display device 10 is brought into close contact with the wall. After aligning the model data 15b with the image data, the control unit 16 performs a process of continuously aligning the model data 15b with the image data based on the position information detected by the positioning unit 13.

[0055] According to embodiment 3, the augmented reality display device 10 is placed close to the corner of the wall corresponding to the virtual AR marker assigned to the model data 15b, thereby aligning the model data 15b with the image data. This contributes to aligning the model data 15b (virtual image) with the image data (real image) even in a location where many obstacles exist or where the location is narrow.

[0056] The augmented reality display devices according to the first to third embodiments can be configured using so-called hardware resources (information processing devices, computers), and may use those having the configuration shown in Fig. 12. For example, the hardware resources 100 include a processor 101, a memory 102, a network interface 103, and the like, which are interconnected by an internal bus 104.

[0057] 12 is not intended to limit the hardware configuration of the hardware resource 100. The hardware resource 100 may include hardware (e.g., an input / output interface) that is not shown. Furthermore, the number of units such as the processor 101 included in the device is not intended to be limited to the example shown in FIG. 12, and for example, multiple processors 101 may be included in the hardware resource 100. The processor 101 may be, for example, a central processing unit (CPU), a microprocessor unit (MPU), a graphics processing unit (GPU), or the like.

[0058] The memory 102 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD).

[0059] The network interface 103 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0060] The functions of the hardware resource 100 are realized by the processing modules described above. The processing modules are realized, for example, by the processor 101 executing a program stored in the memory 102. The programs can be updated by downloading them over a network or by using a storage medium that stores the programs. Furthermore, the processing modules may be realized by semiconductor chips. In other words, it is sufficient that the functions performed by the processing modules be realized by software being executed on some kind of hardware.

[0061] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0062] [Appendix 1] An augmented reality display device that displays model data aligned with image data of a captured object, a display configured to display information; an imaging unit configured to generate image data by photographing an object to be photographed; a positioning unit configured to detect position information of the augmented reality display device; a storage unit configured to store predetermined applications and model data; a control unit configured to control the display unit, the imaging unit, the positioning unit, and the storage unit, and to perform predetermined information processing; Equipped with The model data is provided with a plurality of virtual AR markers, each of which is associated with position information corresponding to one of the corners of a plurality of walls; The control unit a process of displaying a message on the display unit to align the augmented reality display device with a corner of a wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached, and to bring one end surface of the augmented reality display device into close contact with the wall, when the predetermined application is launched; a process of highlighting the nearest virtual AR marker on the model data or the image data on the display unit; a process of aligning the model data with the image data when the one end surface of the augmented reality display device is in close contact with the wall; a process of continuously aligning the model data with the image data based on position information detected by the positioning unit after aligning the model data with the image data; The augmented reality display device is configured to: [Appendix 2] the positioning unit has a function of detecting position information of the augmented reality display device using SLAM or IMU, The control unit is configured to perform a process of continuously aligning the model data and the image data based on the position information of the augmented reality display device detected by the SLAM or IMU. 10. The augmented reality display device of claim 1. [Appendix 3] The control unit is configured to perform processing to display, on the display unit, the virtual AR marker assigned to a location corresponding to a corner of a wall included in the image data while continuously aligning the model data with the image data. 3. The augmented reality display device according to claim 1 or 2. [Appendix 4] The control unit is configured to, when a movement amount or time of the augmented reality display device after being brought into close contact with the wall becomes equal to or greater than a preset threshold, perform processing to align the augmented reality display device with a corner of the wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached, and display a message on the display unit to instruct the augmented reality display device to bring the one end face of the augmented reality display device into close contact with the wall. 4. An augmented reality display device according to any one of claims 1 to 3. [Appendix 5] the control unit is configured to perform processing for highlighting the nearest virtual AR marker on the display unit when a movement amount or time of the augmented reality display device after the augmented reality display device comes into contact with the wall becomes equal to or greater than a preset threshold and a corner of the wall corresponding to the nearest virtual AR marker is included in the image data. 5. The augmented reality display device of claim 4. [Appendix 6] The control unit a process of realigning the model data and the image data when the one end surface of the augmented reality display device is brought into close contact with the wall; a process of continuously aligning the model data with the image data based on position information detected by the positioning unit after aligning the model data with the image data again; configured to: 6. The augmented reality display device according to claim 4 or 5. [Appendix 7] the control unit is configured to measure a distance from the imaging unit to the ground when aligning the model data with the image data during the close contact, and to align the model data with the image data based on the measured distance. 7. An augmented reality display device according to any one of appendices 1 to 6. [Appendix 8] the control unit is configured to measure a distance from the imaging unit to the ground when the surface of the display unit is made horizontal by a level function, and to align the model data with the image data based on the measured distance, when aligning the model data with the image data during the close contact. 8. The augmented reality display device of claim 7. [Appendix 9] the control unit is configured to, when aligning the model data with the image data during the close contact, measure an orientation of the imaging unit and a distance from the imaging unit to the ground, calculate a vertical distance from the imaging unit to the ground based on the measured orientation of the imaging unit and the measured distance from the imaging unit to the ground, and align the model data with the image data based on the calculated distance. 8. The augmented reality display device of claim 7. [Appendix 10] the positioning unit has a function of detecting position information of the augmented reality display device by a predetermined functional unit different from SLAM and IMU, The control unit is configured to, when highlighting the nearest virtual AR marker, display the position and orientation of the augmented reality display device in the model data based on the position information of the augmented reality display device detected by the predetermined function unit and the position information of the nearest virtual AR marker, or to display the model data in accordance with the image data. 10. An augmented reality display device according to any one of Supplementary Notes 1 to 9. [Appendix 11] The control unit is configured to perform processing for displaying the message on the display unit so that, when displaying the message, the other end face of the augmented reality display device, which is perpendicular to the one end face of the augmented reality display device, is aligned with the corner of the wall and the one end face of the augmented reality display device is brought into close contact with the wall. An augmented reality display device according to any one of Supplementary Notes 1 to 10. [Appendix 12] The control unit is configured to perform offset correction of the position of the model data or the image data in accordance with a preset position of the imaging unit in the augmented reality display device when aligning the model data and the image data. 12. An augmented reality display device according to any one of Supplementary Notes 1 to 11. [Appendix 13] further comprising an input unit configured to input information through a user operation; the storage unit is further configured to store VR data corresponding to the image data and positionally associated with the model data, and a database; The control unit a process of storing, in the database, anchor information created by the user operating the input unit to input arbitrary information by specifying an arbitrary position relative to the virtual AR marker that served as the basis for alignment, while continuously aligning the model data with the image data based on the position information detected by the positioning unit; a process of displaying the anchor information at a position in the VR data corresponding to a position specified in the anchor information when the VR data is displayed on the display unit; configured to: 13. An augmented reality display device according to any one of appendices 1 to 12. [Appendix 14] An augmented reality display method for displaying model data by aligning it with image data of a captured object using hardware resources, comprising: The model data is provided with a plurality of virtual AR markers, each of which is associated with position information corresponding to one of the corners of a plurality of walls; a step of displaying a message prompting the user to align the hardware resource with a corner of a wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached and to bring one end surface of the hardware resource into close contact with the wall when a predetermined application is launched; highlighting the nearest virtual AR marker on the model data or the image data; a step of aligning the model data with the image data when the one end surface of the hardware resource is in close contact with the wall; a step of continuously aligning the model data and the image data based on position information detected by positioning after aligning the model data and the image data; An augmented reality display method, including: [Appendix 15] A program that causes hardware resources to execute a process of aligning model data with image data of a photographed object and displaying the aligned model data, The model data is provided with a plurality of virtual AR markers, each of which is associated with position information corresponding to one of the corners of a plurality of walls; a process of displaying a message instructing the user to align the hardware resource with a corner of a wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached and to bring one end face of the hardware resource into close contact with the wall when a predetermined application is launched; highlighting the nearest virtual AR marker on the model data or the image data; a process of aligning the model data with the image data when the one end surface of the hardware resource is in close contact with the wall; a process of continuously aligning the model data and the image data based on position information detected by positioning after aligning the model data and the image data; A program that causes the hardware resource to execute the above.

[0063] The disclosures of the above-cited patent documents are incorporated herein by reference and may be used as the basis or part of the present invention, as necessary. Modifications and adjustments of the embodiments and examples are possible within the scope of the entire disclosure of the present invention (including the claims and drawings), and further based on the basic technical concept thereof. Furthermore, various combinations and selections (or non-selections, as necessary) of the various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of the entire disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concept, including the claims and drawings. Furthermore, with regard to the numerical values ​​and numerical ranges described in this application, any intermediate values, lower values, and smaller ranges are deemed to be included, even if not explicitly stated. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of the disclosure of the present invention, in accordance with the spirit of the present invention, are also deemed to be included in (belong to) the disclosures of this application. [Explanation of symbols]

[0064] 1. Subject of photography 2 models 3, 3a-3e, 3L, 3R Virtual AR marker 4 Reference points 5 Current device location 6 Device orientation 7. Wall 7a Edge 8 moves 9. Anchor 10 Augmented reality display device 11 Display section 12 Imaging unit 13 Positioning unit 14 Input section 15 Storage section 15a AR app (prescribed app) 15b model data 15c VR data 15d database 16 Control Unit 100 Hardware Resources 101 processors 102 memory 103 Network Interface 104 Internal Bus

Claims

1. An augmented reality display device that displays model data aligned with image data of a captured object, a display configured to display information; an imaging unit configured to generate image data by photographing an object to be photographed; a positioning unit configured to detect position information of the augmented reality display device; a storage unit configured to store predetermined applications and model data; a control unit configured to control the display unit, the imaging unit, the positioning unit, and the storage unit, and to perform predetermined information processing; Equipped with a plurality of virtual AR markers are assigned to the model data, the virtual markers being associated with position information corresponding to any of the corners of a plurality of walls; The control unit a process of displaying a message on the display unit to align the augmented reality display device with a corner of a wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached, and to bring one end surface of the augmented reality display device into close contact with the wall, when the predetermined application is started; a process of highlighting the nearest virtual AR marker on the model data or the image data on the display unit; a process of aligning the model data with the image data when the one end surface of the augmented reality display device is in close contact with the wall; a process of continuously aligning the model data with the image data based on position information detected by the positioning unit after aligning the model data with the image data; The augmented reality display device is configured to:

2. the positioning unit has a function of detecting position information of the augmented reality display device using SLAM or IMU, The control unit is configured to perform a process of continuously aligning the model data and the image data based on the position information of the augmented reality display device detected by the SLAM or IMU. The augmented reality display device according to claim 1 .

3. the control unit is configured to perform processing to display, on the display unit, the virtual AR markers assigned to locations corresponding to corners of walls included in the image data while continuously aligning the model data with the image data. The augmented reality display device according to claim 1 or 2.

4. The control unit is configured to, when a movement amount or time of the augmented reality display device after being brought into close contact with the wall becomes equal to or greater than a preset threshold, perform processing to align the augmented reality display device with a corner of the wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached and display a message on the display unit to instruct the augmented reality display device to bring the one end surface of the augmented reality display device into close contact with the wall. The augmented reality display device according to any one of claims 1 to 3.

5. the control unit is configured to perform processing to highlight the nearest virtual AR marker on the display unit when a movement amount or time of the augmented reality display device after the augmented reality display device is brought into close contact with the wall becomes equal to or greater than a preset threshold and a corner of the wall corresponding to the nearest virtual AR marker is included in the image data. The augmented reality display device according to claim 4 .

6. The control unit a process of realigning the model data and the image data when the one end surface of the augmented reality display device is brought into close contact with the wall; a process of continuously aligning the model data with the image data based on position information detected by the positioning unit after aligning the model data with the image data again; configured to: The augmented reality display device according to claim 4 or 5.

7. the control unit is configured to measure a distance from the imaging unit to the ground when aligning the model data with the image data during the close contact, and to align the model data with the image data based on the measured distance. The augmented reality display device according to any one of claims 1 to 6.

8. the control unit is configured to measure a distance from the imaging unit to the ground when the surface of the display unit is made horizontal by a level function, and to align the model data with the image data based on the measured distance, when aligning the model data with the image data during the close contact. The augmented reality display device according to claim 7 .

9. the control unit is configured to, when aligning the model data with the image data during the close contact, measure an orientation of the imaging unit and a distance from the imaging unit to the ground, calculate a vertical distance from the imaging unit to the ground based on the measured orientation of the imaging unit and the measured distance from the imaging unit to the ground, and align the model data with the image data based on the calculated distance. The augmented reality display device according to claim 7 .

10. the positioning unit has a function of detecting position information of the augmented reality display device by a predetermined functional unit different from the SLAM and the IMU, The control unit is configured to, when highlighting the nearest virtual AR marker, display the position and orientation of the augmented reality display device in the model data based on the position information of the augmented reality display device detected by the predetermined function unit and the position information of the nearest virtual AR marker, or to display the model data in accordance with the image data. The augmented reality display device according to any one of claims 1 to 9.

11. The control unit is configured to perform processing for displaying the message on the display unit so that, when displaying the message, the other end face of the augmented reality display device, which is perpendicular to the one end face of the augmented reality display device, is aligned with the corner of the wall and the one end face of the augmented reality display device is brought into close contact with the wall. The augmented reality display device according to any one of claims 1 to 10.

12. The control unit is configured to perform offset correction of the position of the model data or the image data in accordance with a preset position of the imaging unit in the augmented reality display device when aligning the model data and the image data.

12. An augmented reality display device according to any one of claims 1 to 11.

13. further comprising an input unit configured to input information through a user operation; the storage unit is further configured to store VR data corresponding to the image data and positionally associated with the model data, and a database; The control unit a process of storing, in the database, anchor information created by the user operating the input unit to input arbitrary information by specifying an arbitrary position relative to the virtual AR marker that served as the basis for alignment, while continuously aligning the model data with the image data based on the position information detected by the positioning unit; a process of displaying the anchor information at a position in the VR data corresponding to a position specified in the anchor information when the VR data is displayed on the display unit; configured to:

13. An augmented reality display device according to any one of claims 1 to 12.

14. An augmented reality display method for displaying model data by aligning it with image data of a captured object using hardware resources, comprising: a plurality of virtual AR markers are assigned to the model data, the virtual markers being associated with position information corresponding to any of the corners of a plurality of walls; a step of displaying a message instructing the user to align the hardware resource with a corner of a wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached and to bring one end surface of the hardware resource into close contact with the wall when a predetermined application is launched; highlighting the nearest virtual AR marker on the model data or the image data; a step of aligning the model data with the image data when the one end surface of the hardware resource is in close contact with the wall; a step of continuously aligning the model data and the image data based on position information detected by positioning after aligning the model data and the image data; An augmented reality display method, including:

15. A program that causes hardware resources to execute a process of aligning model data with image data of a photographed object and displaying the aligned model data, a plurality of virtual AR markers are assigned to the model data, the virtual markers being associated with position information corresponding to any of the corners of a plurality of walls; a process of displaying a message instructing the user to align the hardware resource with a corner of a wall to which a nearest virtual AR marker among the plurality of virtual AR markers is attached and to bring one end face of the hardware resource into close contact with the wall when a predetermined application is launched; a process of highlighting the nearest virtual AR marker on the model data or the image data; a process of aligning the model data with the image data when the one end surface of the hardware resource is in close contact with the wall; a process of continuously aligning the model data and the image data based on position information detected by positioning after aligning the model data and the image data; A program that causes the hardware resource to execute the above.

Images