Virtual reality display system and virtual reality display method

Abstract

To provide a virtual reality display system and a virtual reality display method which reduce deviation between a virtual image and a picked-up image to be displayed.SOLUTION: A virtual reality display system 1 comprises: an arithmetic unit 6 provided with a first camera 3 which picks up a first marker M1 provided in a building, a second camera 4 integrally provided with the first camera 3, and a first arithmetic logical unit 7; a second marker M2 which is picked up by the second camera 4; and a mobile terminal 10 provided with a third camera 11 which picks up the building, an inertia detection unit 15 which detects change of acceleration and a posture, a second arithmetic logical unit 13, and a display unit 12 which displays a virtual image. The first arithmetic logical unit 7 calculates reference coordinates on the basis of the first marker and calculates data regarding a position and a posture of the mobile terminal to the reference coordinates. The second arithmetic logical unit generates the virtual image obtained by overlapping a virtual three-dimensional object with a picked-up image acquired from the third camera and updates the virtual image on the basis of the change of the acceleration and the posture detected by the inertia detection unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a virtual reality display system and a virtual reality display method for displaying a virtual image. [Background technology]

[0002] Augmented Reality (AR) technology is being researched, which displays a real image and a virtual image by superimposing them on each other. The AR technology detects the position and orientation of the AR terminal, and displays a virtual image by superimposing it on the real image based on the detection result. The AR terminal uses a camera to capture a reference marker such as a two-dimensional code (see, for example, Patent Document 1) placed at a known position and orientation, and acquires the position and orientation of the terminal at that time as initial values ​​based on this. The AR terminal then measures changes in its own position and orientation using an inertial measurement unit (IMU) installed in it, and estimates the current position and orientation by accumulating the changes from the initial values. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-523500 Summary of the Invention [Problem to be solved by the invention]

[0004] When estimating position and orientation using IMU measurements, drift (accumulated error) occurs due to accumulated errors, and a gradual shift occurs between the real image and the virtual image in the AR display. Therefore, the AR device must capture the reference marker as needed to reset the device's position and orientation to the initial values.

[0005] An object of the present invention is to provide a virtual reality display system that can display a virtual image and a captured image with reduced deviation. [Means for solving the problem]

[0006] One aspect of the present invention is a virtual reality display system that generates a virtual image by superimposing a virtual three-dimensional object on an image of a building within the building and displays the virtual image, the system comprising: a position detection device including a first marker provided on the building, a first camera that images the first marker and obtains a first captured image, a second camera that is provided integrally with the first camera, and a first calculation unit; a mobile terminal including a second marker imaged by the second camera, a third camera that images the building and obtains the captured image, an inertial detection unit that detects acceleration and changes in attitude, the second calculation unit, and a display unit that displays the virtual image, and in the position detection device, the first calculation unit calculates the first marker recorded in the first captured image by the first calculation unit. a reference coordinate in the building based on a second marker recorded in a second captured image acquired by the second camera, and calculates data regarding the position and attitude of the mobile terminal relative to the reference coordinate based on the reference coordinate and a second marker recorded in a second captured image acquired by the second camera; and in the mobile terminal, the second calculation unit generates the virtual image by superimposing a virtual three-dimensional object on the captured image acquired from the third camera based on the data regarding the position and attitude acquired from the position detection device at a first sampling rate, and updates the virtual image based on changes in acceleration and attitude detected by the inertial detection unit at a second sampling rate higher than the first sampling rate.

[0007] According to the present invention, a second marker provided on a mobile terminal is imaged by a second camera, and the current position and attitude of the mobile terminal are calculated by a computing device, thereby reducing the deviation due to the accumulated error in the position and attitude between the captured image and the virtual structure that is set based on the detection results of the inertial detection unit.

[0008] One aspect of the present invention is a virtual reality display method for generating a virtual image in which a virtual three-dimensional object is superimposed on an image of the building and displaying the virtual image, the virtual image having a mobile terminal including a first marker provided on a building, a first camera that images the first marker and obtains a first captured image, and a second camera that is provided integrally with the first camera, the second marker imaged by the second camera, a third camera that images the building and obtains the captured image, an inertial detection unit that detects changes in acceleration and attitude, and a display unit, the method including: calculating a reference coordinate in the building based on the first marker recorded in the first captured image; The virtual reality display method includes the steps of: calculating data regarding the position and attitude of the mobile terminal relative to the reference coordinates based on the reference coordinates and a second marker recorded in a second captured image acquired by the second camera; generating the virtual image by superimposing a virtual three-dimensional object on the captured image acquired from the third camera based on the data regarding the position and attitude acquired from the position detection device at a first sampling rate; updating the virtual image based on changes in acceleration and attitude detected by the inertial detection unit at a second sampling rate higher than the first sampling rate; and displaying the virtual image on a display unit.

[0009] According to the present invention, it is possible to generate a virtual image in which the positional deviation between a captured image and a virtual three-dimensional object displayed on a mobile terminal is reduced. Effect of the Invention

[0010] According to the present invention, it is possible to reduce the deviation between a virtual image and a captured image when the virtual image is displayed on a mobile terminal. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating a schematic configuration of a virtual reality display system according to an embodiment. [Diagram 2] 1 is a block diagram showing the configuration of a virtual reality display system. [Diagram 3]13 is a diagram showing reference coordinates of a first marker calculated based on a first captured image. FIG. [Figure 4] 13 is a diagram illustrating a method for calculating position and orientation data of a mobile terminal based on a second captured image of a second marker. FIG. [Diagram 5] FIG. 13 is a diagram illustrating a method for calculating the coordinates of the position and attitude of a mobile terminal. [Figure 6] FIG. 13 is a diagram illustrating an example of a virtual image. [Figure 7] 13 is a flowchart showing a process for generating a virtual image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of a virtual reality display system according to the present invention will be described with reference to the drawings. The virtual reality display system mobile terminal is a system that displays a virtual image in which a virtual three-dimensional object is superimposed on a captured image of a building.

[0013] As shown in FIG. 1, the virtual reality display system 1 includes a first marker M1 provided on the floor of a building, a position detection device 2 installed in the building, and a mobile terminal 10 capable of communicating with the position detection device 2.

[0014] The first marker M1 is, for example, a two-dimensional code. The first marker M1 may be formed by other patterns as long as the position, posture, and information can be recognized. The first marker M1 may be a two-dimensional code printed on a sheet body, or may be a two-dimensional code printed directly on the floor surface. The first marker M1 is installed at an arbitrary position on the floor surface. The first marker M1 is installed, for example, so that one side is aligned with the direction of the wall of the building. The position detection device 2 includes, for example, a tripod 5 installed on the floor surface, a first camera 3, a second camera 4, and a computing device 6 connected to the first camera 3 and the second camera 4, and is installed vertically above the first marker M1. The tripod 5 has a base 5A and a leg 5B that extends vertically downward from the base 5A and is grounded on the floor surface. The first camera 3 is fixed to the base 5A of the tripod 5 in a state facing vertically downward so that the vertical direction is the imaging direction, and is installed below the base 5A in this embodiment. The second camera 4 is fixed to the base 5A of the tripod 5 in a state facing horizontally so that the imaging direction is perpendicular to the imaging direction of the first camera 3, that is, the horizontal direction, and in this embodiment, is provided above the base 5A. The first camera 3 and the second camera 4 are integrated so that their relative positions do not change, and in this embodiment, they are fixed to the same base 5A so that their relative positions do not change.

[0015] The calculation device 6 is communicatively connected to the first camera 3 and the second camera 4, for example, by wire or wirelessly. The calculation device 6 includes, for example, a first calculation unit 7, a first storage unit 8, and a first communication unit 9. The first calculation unit 7 generates a virtual image based on acquired data. The first storage unit 8 stores data and programs used in the calculation of the first calculation unit 7, and is, for example, a storage medium such as a HDD or a flash memory. The first communication unit 9 is a communication interface that communicates with the mobile terminal 10. The first communication unit 9 is capable of communicating with the mobile terminal 10 using a communication means such as a wireless LAN, Bluetooth (registered trademark), or a mobile communication network.

[0016] The mobile terminal 10 is a portable terminal device such as a smartphone, a tablet terminal, a personal computer, etc. The mobile terminal 10 includes, for example, a third camera 11, a display unit 12 provided on the front side, an inertia detection unit 15, a second calculation unit 13, a second storage unit 14, a second communication unit 16, and a second marker M2.

[0017] The third camera 11 is an electronic camera device that captures an image of a building and obtains the captured image. The display unit 12 displays the captured image and the virtual image, and is, for example, a display device configured with a liquid crystal display, an organic EL display, or the like. The inertial detection unit 15 is a three-axis acceleration sensor or a three-axis angular velocity sensor that detects three-axis angles or angular velocities according to the current acceleration and changes in attitude, and is, for example, an inertial measurement unit (IMU). The inertial detection unit 15 uses a high frequency sensor, for example, a sampling rate of 100 Hz or more, and in this embodiment, a sampling rate of 100 Hz is used. The second calculation unit 13 generates a virtual image in which a virtual three-dimensional object is superimposed on the captured image. The second storage unit 14 is, for example, a storage medium such as an HDD or a flash memory that stores data and programs required for calculation.

[0018] The second communication unit 16 is a communication interface that communicates with the position detection device 2 (first communication unit 9). The first communication unit 9 is capable of communicating with the position detection device 2 using a communication means such as wireless LAN, Bluetooth (registered trademark), or a mobile communication network. The second marker M2 is a two-dimensional code or the like provided on the rear side of the mobile terminal 10. The second marker M2 may be a two-dimensional code printed on a sheet body or may be printed directly on the rear side of the mobile terminal 10. The second marker M2 may be formed by another pattern as long as the position, posture, and information can be recognized.

[0019] Next, a description will be given of the processing of the virtual reality display system 1. First, in the position detection device 2, the first camera 3 captures an image of a first marker M1 provided on the floor of a building. The first camera 3 outputs a first captured image G1 in which the first marker M1 is recorded to the calculation device 6.

[0020] In the calculation device 6, the first calculation unit 7 extracts an image area of ​​the first marker M1 from the first captured image G1. The first calculation unit 7, for example, recognizes a finder pattern P that is a feature point for detecting the position of three corners provided on the first marker M1, and extracts the first marker M1. The first calculation unit 7 reads information on the first marker M1 based on the pattern recorded on the first marker M1, and recognizes the ID of the first marker M1.

[0021] The first calculation unit 7 calculates the relative position between the first camera 3 and the first marker M1 based on the ratio of the image area of ​​the first marker M1 to the first captured image G1. The relative position is calculated based on, for example, the ratio of the image area of ​​the first marker M1 in the first captured image G1 captured in advance at a predetermined distance. The first calculation unit 7 calculates the relative angle between the first camera 3 and the first marker M1 based on the inclination of the image area of ​​the first marker M1 in the first captured image G1. Based on the calculation result, the first calculation unit 7 sets a three-dimensional coordinate system in the building with the center of the first marker M1 as the reference coordinate (origin).

[0022] Next, in the position detection device 2, the second camera 4 captures an image of the second marker M2 provided on the mobile terminal 10. The second camera 4 outputs to the calculation device 6 a second captured image G2 in which the second marker M2 is recorded.

[0023] The first calculation unit 7 extracts the image area of ​​the second marker M2 from the second captured image G2. The first calculation unit 7 recognizes, for example, a finder pattern P that is a feature point for detecting the position of three corners provided on the second marker M2, and extracts the second marker M2. The first calculation unit 7 obtains the ID and recording information of the second marker M2 from the pattern recorded in the second marker M2.

[0024] The first calculation unit 7 also extracts the image area of ​​the second marker M2 in the second captured image G2, and calculates the relative coordinates and relative orientation between the second camera 4 and the mobile terminal 10 based on the sizes and ratios of the four sides (a, b, c, d in the figure) of the image area of ​​the second marker M2. The first calculation unit 7 sets, for example, the center of the image area of ​​the second marker M2 as the coordinates of the mobile terminal 10.

[0025] As shown in FIG. 5, the first calculation unit 7 calculates position and orientation data (data related to position and orientation) including the coordinates (x, y, z) and orientation (α, β, γ) of the mobile terminal 10 relative to the reference coordinates based on the calculation results of the relative coordinates and relative orientation between the second camera 4 and the mobile terminal 10, the reference coordinates, the coordinates of the first camera 3, and the coordinates of the second camera 4. The coordinates of the first camera 3 and the coordinates of the second camera 4 are the positions of the first camera 3 and the second camera 4 relative to the reference coordinates, respectively. The first calculation unit 7 recognizes the mobile terminal 10 based on the ID acquired from the second marker M2, and transmits the calculation results to the mobile terminal 10 (second communication unit 16) via the first communication unit 9 based on the recording information acquired from the second marker M2. The first calculation unit 7 transmits the position and orientation data to the mobile terminal 10 at a predetermined timing at a predetermined time interval, for example, at a sampling rate of 10 Hz.

[0026] In the mobile terminal 10, the second calculation unit 13 updates the position and attitude data of the mobile terminal 10 at the timing (first sampling rate) when the position and attitude data is received from the calculation device 6 of the position detection device 2. Thereafter, the second calculation unit 13 calculates the amount of change in coordinates and attitude based on the change in acceleration and attitude detected by the inertial detection unit 15, adds the amount of change to the received position and attitude data, and calculates new position and attitude data. In other words, if a change in position and attitude occurs since the acquisition of the position and attitude data, the second calculation unit 13 updates the position and attitude data of the mobile terminal 10 based on the position and attitude data and the detection data of the inertial detection unit 15.

[0027] As shown in FIG. 6, the second calculation unit 13 generates an image of a virtual solid object G whose position, orientation, and size are adjusted based on the position and orientation data received from the calculation unit 6 of the position detection device 2. The virtual solid object G is an image of an object that does not exist in the current building, such as a structure or equipment to be installed in the building. The second calculation unit 13, for example, identifies a structure B of the building displayed in the current captured image based on the position and orientation data, and extracts a virtual solid object G to be added to the image of the structure B. The second calculation unit 13 generates an image of the virtual solid object G whose position, orientation, and size are adjusted based on the current position and orientation data. The second calculation unit 13 overlays the generated image of the virtual solid object G on the structure B in the captured image to generate a virtual image V.

[0028] When the second calculation unit 13 acquires new position and orientation data from the calculation device 6 of the position detection device 2 at a timing based on a predetermined sampling rate (first sampling rate), it adjusts the position of the virtual three-dimensional object G based on the new position and orientation data and generates a new virtual image V.

[0029] Furthermore, in this embodiment, in addition to when new position and orientation data is acquired from the calculation device 6 of the position detection device 2, the second calculation unit 13 updates the position and orientation data based on the change in acceleration and orientation detected by the inertial detection unit 15 outputted at the second sampling rate in conjunction with a change in the position and orientation of the mobile terminal 10 by the user, and updates the virtual image V. When new position and orientation data is received from the calculation device 6 of the position detection device 2, the second calculation unit 13 updates the current position and orientation data, and generates a new virtual image V based on the updated position and orientation data.

[0030] In this way, the second calculation unit 13 can generate a virtual image V with high accuracy and period by using the detection value of the inertial detection unit 15 outputted at a high sampling rate (second sampling rate, 100 Hz or more) which is low in accuracy, and the detection value of the position and orientation data based on the second marker M2 outputted at a low sampling rate (first sampling rate, about 10 Hz or less) which is high in accuracy. In other words, by making the second sampling rate of the inertial detection unit 15 higher than the first sampling rate which corresponds to the detection period of the position and orientation data based on the second marker M2, it is possible to reduce the restriction of the moving range and orientation of the mobile terminal 10 while suppressing the decrease in accuracy due to the influence of drift which becomes a problem when the inertial detection unit 15 is used. Therefore, the second calculation unit 13 can constantly generate a virtual image V with reduced deviation in the position and orientation of the virtual three-dimensional object G by repeating the above calculation.

[0031] By processing the virtual reality display system 1 in this manner, the position and orientation data of the virtual image V is updated based on the second captured image G2 of the second marker M2 before the accumulated errors based on the acceleration and attitude changes detected by the inertial detection unit 15 increase, thereby suppressing the deviation between the virtual structure in the virtual image V and the image of the actual structure B.

[0032] In this embodiment, the virtual image V is displayed on the display unit 12 of the mobile terminal 10 so that the user can check the virtual three-dimensional object G. Since the screen of the virtual image V is updated based on the frame rate, it is preferable that the second calculation unit 13 generates the virtual image V at a frame rate of, for example, 30 Hz or more. In particular, since the update of the position and orientation data based on the second marker M2 has a low period (first sampling rate, about 10 Hz) compared to the frame rate (30 Hz or more), a large delay occurs with respect to the change in the position and orientation of the mobile terminal 10 when the virtual image V is updated based on the update of the position and orientation data based on the second marker M2, and it is not possible to display the virtual image V that smoothly follows the change in the position and orientation of the mobile terminal 10 on the display unit 12 of the mobile terminal 10. However, in this embodiment, the virtual image V is updated based on the detection value of the inertial detection unit 15 that is output at a higher frequency (second sampling rate, 100 Hz or more) compared to the frame rate (30 Hz or more), so that a virtual image V that follows changes in the position and attitude of the mobile terminal 10 with less delay and more smoothly can be displayed on the display unit 12 of the mobile terminal 10.

[0033] Next, each step for generating a virtual image in the virtual reality display system 1 will be described.

[0034] 7 shows a flow chart of each process of the virtual reality display method for generating a virtual image. The mobile terminal 10 captures an image of a building (step S10). The first camera 3 captures an image of a first marker M1 provided on the building (step S12). The second camera 4 captures an image of a second marker M2 provided on the mobile terminal 10 (step S14). The arithmetic device 6 calculates a reference coordinate in the building based on the first marker M1 recorded in the first captured image G1 (step S16). The arithmetic device 6 calculates position and orientation data of the mobile terminal 10 relative to the reference coordinate based on the reference coordinate and the second marker M2 recorded in the second captured image G2 (step S18).

[0035] In the mobile terminal 10, a virtual three-dimensional object G is generated based on the position and orientation data of the mobile terminal 10 based on the calculation result of the computing device 6 at the first sampling rate, and a virtual image V is generated by superimposing the virtual three-dimensional object G on the captured image (step S20). The virtual image V is updated based on the acceleration and the change in the orientation detected by the inertial detection unit 15 at a second sampling rate higher than the first sampling rate (step S22). The virtual image V is displayed on the display unit 12 (step S24).

[0036] As described above, the virtual reality display system 1 can generate a virtual image in which the positional deviation between the captured image and a virtual three-dimensional object is reduced. The virtual reality display system 1 can reduce deviations based on accumulated errors in the position and attitude between the captured image and the virtual structure, which are set based on changes in acceleration and attitude detected by the inertial detection unit 15, by capturing an image of the second marker M2 provided on the mobile terminal 10 with the second camera 4 and calculating the current position and attitude of the mobile terminal 10 with a computing device.

[0037] The above-mentioned first calculation unit 7 and second calculation unit 13 are realized by a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) executing a program (software). Some or all of these functional units may be realized by hardware such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or a flash memory included in the first storage unit 8 and the second storage unit 14, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the storage device by mounting the storage medium in a drive device.

[0038] Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and can be modified as appropriate without departing from the spirit of the present invention. For example, the arithmetic device 6 may be integrated into the mobile terminal 10. The arithmetic device 6 may be a server connected to be able to communicate through a network. [Explanation of symbols]

[0039] 1 Virtual reality display system 3. First Camera 4. Second Camera 6 Arithmetic unit 10 Mobile terminals 12 Display section G1 First captured image G2 Second captured image M1 First marker M2 Second marker G Virtual Object V Virtual Image

Claims

1. A virtual reality display system that generates a virtual image by superimposing a virtual three-dimensional object on an image of a building captured in the building, and displays the virtual image, A first marker provided on the building; a position detection device including a first camera that captures an image of the first marker and obtains a first captured image, a second camera that is integral with the first camera, and a first calculation unit; a mobile terminal including: a second marker captured by the second camera; a third camera that captures an image of the building and acquires the captured image; an inertial detection unit that detects a change in acceleration and attitude; a second calculation unit; and a display unit that displays the virtual image; having In the position detection device, the first calculation unit calculates a reference coordinate in the building based on the first marker recorded in the first captured image, and calculates data on a position and attitude of the mobile terminal with respect to the reference coordinate based on the reference coordinate and a second marker recorded in a second captured image acquired by the second camera; In the mobile terminal, the second calculation unit generates the virtual image by superimposing a virtual three-dimensional object on the captured image acquired from the third camera based on the data regarding the position and attitude acquired from the position detection device at a first sampling rate, and updates the virtual image based on changes in acceleration and attitude detected by the inertial detection unit at a second sampling rate higher than the first sampling rate. A virtual reality display system comprising:

2. A first marker provided on the building; a position detection device including a first camera that captures an image of the first marker and obtains a first captured image, and a second camera that is integral with the first camera; a mobile terminal including: a second marker captured by the second camera; a third camera capturing an image of the building and acquiring the captured image; an inertial detection unit detecting acceleration and changes in attitude; and a display unit; A virtual reality display method for generating a virtual image in which a virtual three-dimensional object is superimposed on a captured image of the building in the building, and displaying the virtual image, comprising: calculating a reference coordinate in the building based on the first marker recorded in the first captured image; calculating data on a position and an attitude of the mobile terminal with respect to the reference coordinates based on the reference coordinates and a second marker recorded in a second captured image acquired by the second camera; generating the virtual image by superimposing a virtual three-dimensional object on the captured image acquired from the third camera based on the data regarding the position and orientation acquired from the position detection device at a first sampling rate; updating the virtual image based on changes in acceleration and attitude detected by the inertial detection unit at a second sampling rate higher than the first sampling rate; displaying the virtual image on a display unit; A virtual reality display method comprising:

Images