Virtual object display device, virtual object display program, and virtual object display method

Abstract

The virtual object can be displayed after being aligned with the real space without using a marker.SOLUTION: An information processing device includes a gesture detection unit 102 that detects a gesture, a reference position determination unit 103 that determines a position of a reference point in a real space on the basis of a detection result by the gesture detection unit 102, and a virtual object display unit 106 that displays a virtual object after positioning the virtual object with respect to the real space on the basis of the position of the reference point determined by the reference position determination unit 103.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a virtual object display device that displays a virtual object, a virtual object display program, and a virtual object display method. [Background technology]

[0002] Conventionally, in AR or MR, when a virtual object is superimposed on real space, a marker is placed in the real space, and by recognizing this marker, the position of the marker is detected as a reference point, and the virtual object is aligned with this reference point before being displayed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 151493 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when markers are used for alignment, even a slight deviation in the position or angle of the marker affects the alignment between the real space and the virtual object. Therefore, as shown in FIG. 14, even if the position is correct near a reference marker, the error increases as the object moves away from the marker, resulting in a positional deviation. This is particularly likely to affect alignment when the virtual object is large. In FIG. 14, reference numeral 141 denotes an object existing in the real space, reference numeral 142 denotes a virtual object, and reference numeral 143 denotes a marker.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a virtual object display device that can display virtual objects after aligning them with real space without using markers. [Means for solving the problem]

[0006] The virtual object display device according to the present disclosure is characterized by including a gesture detection unit that detects gestures, a reference position determination unit that determines the position of a reference point in real space based on the detection result by the gesture detection unit, and a virtual object display unit that aligns a virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit and then displays the virtual object. [Effects of the Invention]

[0007] According to the present disclosure, with the above configuration, it is possible to display a virtual object after aligning it with real space without using a marker. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a virtual object display device according to a first embodiment. [Figure 2] 5 is a flowchart showing an example of the operation of the virtual object display device according to the first embodiment. [Figure 3] 4 is a diagram illustrating an example of the operation of the virtual object display device according to the first embodiment. FIG. [Figure 4] 5 is a diagram illustrating an example of the operation of a reference position determining unit in the first embodiment. FIG. [Figure 5] 5 is a diagram illustrating an example of the operation of a reference position determining unit in the first embodiment. FIG. [Figure 6] 5 is a diagram illustrating an example of the operation of a reference angle determination unit according to the first embodiment. FIG. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of a virtual object display device according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of the operation of the virtual object display device according to the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of the operation of the virtual object display device according to the second embodiment. [Figure 10]FIG. 10 is a diagram illustrating an example of the configuration of a virtual object display device according to a third embodiment. [Figure 11] 11 is a flowchart showing an example of the operation of the virtual object display device according to the third embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of the operation of the virtual object display device according to the third embodiment. [Figure 13] 13A and 13B are diagrams illustrating examples of hardware configurations of virtual object display devices according to embodiments 1-3. [Figure 14] FIG. 1 is a diagram illustrating an example of displaying a virtual object by a conventional system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of a virtual object display device 1 according to the first embodiment. The virtual object display device 1 displays a virtual object superimposed on a real space. The virtual object display device 1 is applicable to both AR and MR. As shown in FIG. 1 , the virtual object display device 1 according to the first embodiment includes a storage unit 101, a gesture detection unit 102, a reference position determination unit 103, a reference angle determination unit 104, a horizontal direction detection unit 105, and a virtual object display unit 106.

[0010] The storage unit 101 stores various data handled by the virtual object display device 1. For example, data indicating a virtual object is stored in storage unit 101. Note that in the first embodiment, the position of one reference point, the angle of a reference line, and the horizontal direction are set for the virtual object. Furthermore, for example, the storage unit 101 stores data indicating the position of one reference point in real space determined by the reference position determination unit 103. Furthermore, for example, the storage unit 101 stores data indicating the angle of the reference line in real space determined by the reference angle determination unit 104. Furthermore, for example, the storage unit 101 stores data indicating the horizontal direction in real space detected by the horizontal direction detection unit 105.

[0011] Here, examples of the storage unit 101 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, or DVD (Digital Versatile Disc).

[0012] 1 shows a case where the storage unit 101 is provided inside the virtual object display device 1. However, the present invention is not limited to this, and the storage unit 101 may be provided outside the virtual object display device 1.

[0013] The gesture detection unit 102 detects a gesture. In the following, the gesture detection unit 102 will be described mainly as detecting gestures made with the hands, but this is not limiting. For example, the gesture detection unit 102 may detect gestures made with other parts of the body, such as the feet or gaze.

[0014] The reference position determination unit 103 determines the position of a reference point in real space based on the detection result by the gesture detection unit 102. The reference position determination unit 103 in the first embodiment determines the position of one reference point in real space based on the detection result by the gesture detection unit 102. The position of the reference point determined by the reference position determination unit 103 is a position to be aligned with the position of a reference point set for a virtual object.

[0015] For example, when the gesture becomes stationary, the reference position determination unit 103 displays a mark at the gesture position in real space, and when the gesture remains stationary (remains within a predetermined range) after a certain period of time has passed, the reference position determination unit 103 determines the gesture position as the position of the reference point. At this time, the reference position determining unit 103 may reduce the size of the mark over time so that the mark disappears after a certain time has passed.

[0016] Note that the user knows in advance which part of the virtual object the position of one reference point determined by the reference position determination unit 103 will be aligned with.

[0017] Furthermore, the reference position determination unit 103 may be configured to be able to re-determine the position of the reference point after the position of the reference point has been determined based on the detection result by the gesture detection unit 102. At this time, the reference position determination unit 103 may display a bounding box at the position of the reference point in real space, and move the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit 102. Note that, for example, the center position of the bounding box is aligned with the position of the reference point determined by the reference position determination unit 103.

[0018] For example, when the gesture position detected by the gesture detection unit 102 is located inside the bounding box, the reference position determination unit 103 may cancel the determination of the position of the reference point and allow it to be determined again. Also, for example, the bounding box may have an arrow indicating the direction of movement, and the reference position determination unit 103 may move the position of the reference point in the direction of movement indicated by the arrow in accordance with a gesture (for example, an action of pinching the arrow with a hand and moving it) made with respect to the arrow in the bounding box detected by the gesture detection unit 102.

[0019] The reference angle determination unit 104 determines the angle of the reference line in real space based on the detection result by the gesture detection unit 102. The angle of the reference line determined by this reference angle determination unit 104 is a position to be matched with the angle of the reference line set for the virtual object.

[0020] For example, when the gesture is stationary, the reference angle determination unit 104 displays a reference line including the gesture position in real space and the position of the reference point determined by the reference position determination unit 103, and when the gesture remains stationary after a certain period of time has passed (when it remains within a predetermined range), it determines the angle of the reference line. In this case, the reference angle determination unit 104 may display two parallel lines sandwiching a reference line when the gesture is stationary, and may narrow the distance between the two parallel lines over time so that the two parallel lines overlap the reference line after a certain period of time has passed.

[0021] Furthermore, the reference angle determination unit 104 may be configured to be able to re-determine the angle of the reference line after the angle of the reference line has been determined based on the detection result by the gesture detection unit 102. For example, when the gesture position detected by the gesture detection unit 102 is positioned on the reference line, the reference angle determination unit 104 may cancel the determination of the angle of the reference line and allow it to be determined again.

[0022] The horizontal direction detection unit 105 detects the horizontal direction in real space. The horizontal direction detected by the horizontal direction detection unit 105 is a direction to be matched with the horizontal direction set for the virtual object. As the horizontal direction detection unit 105, a sensor capable of recognizing space, which is generally mounted on the virtual object display device 1, can be used.

[0023] The virtual object display unit 106 displays a virtual object after aligning it with the real space based on the position of the reference point determined by the reference position determination unit 103. The virtual object display unit 106 in the first embodiment displays a virtual object after aligning it with the real space based on the position of one reference point determined by the reference position determination unit 103, the angle of the reference line determined by the reference angle determination unit 104, and the horizontal direction detected by the horizontal direction detection unit 105. That is, the virtual object display unit 106 aligns the position of one reference point, the angle of the reference line, and the horizontal direction set for the virtual object stored in the memory unit 101 with the position of one reference point determined by the reference position determination unit 103, the angle of the reference line determined by the reference angle determination unit 104, and the horizontal direction detected by the horizontal direction detection unit 105, and then displays the virtual object in real space.

[0024] Next, an example of the operation of the virtual object display device 1 according to the first embodiment shown in Fig. 1 will be described with reference to Fig. 2. Note that the gesture detection unit 102 constantly detects gestures. In an example of operation of the virtual object display device 1 according to the first embodiment shown in FIG. 1, as shown in FIG. 2, for example, the reference position determination unit 103 first determines the position of one reference point in the real space based on the detection result by the gesture detection unit 102 (step ST101).

[0025] For example, when the gesture becomes stationary, the reference position determination unit 103 displays a mark at the gesture position in real space, and when the gesture remains stationary (remains within a predetermined range) after a certain period of time has passed, the reference position determination unit 103 determines the gesture position as the position of the reference point. At this time, the reference position determining unit 103 may reduce the size of the mark over time so that the mark disappears after a certain time has passed.

[0026] Next, the reference angle determination unit 104 determines the angle of the reference line in real space based on the detection result by the gesture detection unit 102 (step ST102).

[0027] For example, when the gesture is stationary, the reference angle determination unit 104 displays a reference line including the gesture position in real space and the reference point determined by the reference position determination unit 103, and when the gesture remains stationary after a certain period of time has passed (remains within a predetermined range), it determines the angle of the reference line. In this case, the reference angle determination unit 104 may display two parallel lines sandwiching a reference line when the gesture is stationary, and may narrow the distance between the two parallel lines over time so that the two parallel lines overlap the reference line after a certain period of time has passed.

[0028] Furthermore, the horizontal direction detection unit 105 detects the horizontal direction in the real space (step ST103).

[0029] Next, the virtual object display unit 106 aligns the virtual object with the real space based on the position of one reference point determined by the reference position determination unit 103, the angle of the reference line determined by the reference angle determination unit 104, and the horizontal direction detected by the horizontal direction detection unit 105, and then displays the virtual object (step ST104). That is, the virtual object display unit 106 aligns the position of one reference point, the angle of the reference line, and the horizontal direction set for the virtual object stored in the memory unit 101 with the position of one reference point determined by the reference position determination unit 103, the angle of the reference line determined by the reference angle determination unit 104, and the horizontal direction detected by the horizontal direction detection unit 105, and then displays the virtual object in real space.

[0030] Furthermore, the reference position determination unit 103 may be configured to be able to re-determine the position of the reference point after the position of the reference point has been determined based on the detection result by the gesture detection unit 102. At this time, the reference position determination unit 103 may display a bounding box at the position of the reference point in real space, and move the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit 102. Note that, for example, the center position of the bounding box is aligned with the position of the reference point determined by the reference position determination unit 103.

[0031] For example, when the gesture position detected by the gesture detection unit 102 is located inside the bounding box, the reference position determination unit 103 may cancel the determination of the position of the reference point and allow it to be determined again. Furthermore, for example, the bounding box may have an arrow indicating the direction of movement, and the reference position determination unit 103 may move the position of the reference point in the direction of movement indicated by the arrow in accordance with a gesture (for example, an action of pinching the arrow with a hand and moving it) made with respect to the arrow in the bounding box detected by the gesture detection unit 102.

[0032] Furthermore, the reference angle determination unit 104 may be configured to be able to re-determine the angle of the reference line after the angle of the reference line has been determined based on the detection result by the gesture detection unit 102. For example, when the gesture position detected by the gesture detection unit 102 is positioned on the reference line, the reference angle determination unit 104 may cancel the determination of the angle of the reference line and allow it to be determined again.

[0033] Next, an example of the operation of the virtual object display device 1 according to the first embodiment will be described with reference to Fig. 3 to Fig. 6. In the following example, it is assumed that the gesture detection unit 102 detects a gesture made by a hand.

[0034] 3A, first, the reference position determination unit 103 determines the position of one reference point in real space based on the hand gesture detected by the gesture detection unit 102. In the figure, reference numeral 31 denotes an object existing in real space, and reference numeral 32 denotes a virtual object.

[0035] More specifically, for example, first, the user stops their hand at a position to be used as the reference point in real space, as shown in Fig. 4A. In the example of Fig. 4A, the user places their hand on the corner of an object in front of them in real space. In this case, as shown in, for example, FIGS. 3A and 4B, the reference position determination unit 103 displays a mark at the gesture position in real space (in the illustrated example, a corner of a foreground object existing in real space). In the figures, reference numeral 41 indicates the mark. In this case, as shown in FIG. 4, the reference position determination unit 103 may shrink the mark over time so that the mark disappears after a certain period of time has elapsed. FIGS. 3 and 4 show a case in which a circular mark is displayed as an example of the mark. In this way, in the virtual object display device 1 according to embodiment 1, by displaying the mark, the user can easily confirm the position of the reference point. Furthermore, in the virtual object display device 1 according to embodiment 1, by shrinking the mark over time, the user can easily grasp the timing at which the position of the reference point is confirmed. Then, as shown in FIG. 4C, if the gesture remains stationary (remains within a predetermined range) even after a certain period of time has elapsed, the reference position determination unit 103 determines the gesture position as the position of the reference point.

[0036] 3 and 4, the virtual object display unit 106 performs display before the position of the reference point is determined by the reference position determination unit 103. In this case, the virtual object display unit 106 displays the virtual object in accordance with the position of the reference point before determination and the angle of the temporary reference line. Furthermore, when the position of the reference point before determination moves based on the detection result by the gesture detection unit 102, the virtual object display unit 106 changes the position of the virtual object accordingly. In this way, in the virtual object display device 1 according to embodiment 1, by displaying the virtual object before the reference point position is determined by the reference position determination unit 103, the user can perform the operation of determining the reference point position while checking the display state of the virtual object, making it easier to determine the reference point position.

[0037] Then, as shown in FIGS. 3B and 3C, the reference angle determination unit 104 determines the angle of the reference line in real space based on the hand gesture detected by the gesture detection unit 102.

[0038] More specifically, for example, first, after determining the position of one reference point in real space, the user stops one hand at an arbitrary position on the line serving as the reference line in real space, as shown in Fig. 5A. Note that Fig. 5A shows a case in which one hand is stopped at the position of one reference point in real space and the other hand is stopped at an arbitrary position on the line serving as the reference line in real space, but after determining the position of one reference point in real space, it is not necessary to stop one hand and the other hand may be released. As a result, as shown in FIGS. 3B, 3C, and 5, the reference angle determination unit 104 displays a reference line including the position of one reference point and the gesture position in real space, as well as two parallel lines sandwiching the reference line. In the figures, reference numeral 51 denotes the reference line, and reference numeral 52 denotes the parallel lines. Note that the reference line and the two parallel lines displayed by the reference angle determination unit 104 are movable according to the gesture position before the angle of the reference line is determined. Also, as shown in FIG. 4, the reference angle determination unit 104 may narrow the gap between the two parallel lines over time so that the two parallel lines overlap the reference line after a certain period of time has elapsed. In this way, the virtual object display device 1 according to embodiment 1 displays the reference line and the two parallel lines, making it easier for the user to confirm the angle of the reference line. Furthermore, the virtual object display device 1 according to embodiment 1 narrows the gap between the two parallel lines over time, making it easier for the user to grasp the timing at which the angle of the reference line is determined. 3C, the user fine-tunes the angle of the reference line by moving their hand so that the reference line is aligned with the wall surface of an object in real space. At this time, by fine-tuning the reference line so that it is aligned with the wall surface of an object in real space at a location away from the position of one reference point, it is possible to reduce errors. Then, as shown in FIGS. 3C and 5C, reference angle determination unit 104 determines the angle of the reference line when the gesture remains stationary even after a certain period of time has elapsed.

[0039] 3 and 5, the virtual object display unit 106 starts displaying the virtual object before the angle of the reference line is determined by the reference angle determination unit 104. In this case, the virtual object display unit 106 displays the virtual object in accordance with the position of the reference point after determination and the angle of the reference line before determination. Furthermore, when the angle of the reference line before determination moves based on the detection result by the gesture detection unit 102, the virtual object display unit 106 changes the angle of the virtual object accordingly. In this way, in the virtual object display device 1 according to embodiment 1, by displaying a virtual object before the angle of the reference line is determined by the reference angle determination unit 104, the user can perform the operation of determining the angle of the reference line while checking the display state of the virtual object, making it easier to determine the angle of the reference line.

[0040] Furthermore, as shown in FIG. 6, in the virtual object display device 1 according to the first embodiment, after the reference point position is determined by the reference position determining unit 103, the position of the reference point can be determined again.

[0041] 6A, when the gesture position detected by the gesture detection unit 102 is located inside the bounding box, the reference position determination unit 103 cancels the determination of the position of the reference point and enables it to be re-determined. In the figure, reference numeral 61 denotes the bounding box. That is, after the position of the reference point has been determined by the reference position determination unit 103, if the user places his / her hand inside the bounding box displayed at the position of the reference point, the reference position determination unit 103 cancels the determination of the position of the reference point and enables the position of the reference point to be determined again. The method of determining the position of the reference point by the reference position determination unit 103 thereafter is as described above.

[0042] 6B, the bounding box has an arrow indicating the movement direction, and reference position determination unit 103 moves the position of the reference point in the movement direction indicated by the arrow in response to a gesture (for example, an action of grasping the arrow with a hand and moving it) made with respect to the arrow on the bounding box detected by gesture detection unit 102. In the figure, reference numeral 62 indicates an arrow provided on the bounding box. In the example of FIG. 6B, arrows corresponding to each of the three-dimensional directions are added to the bounding box. That is, after the position of the reference point has been determined by the reference position determination unit 103, if the user pinches and moves the arrow added to the bounding box displayed at the position of the reference point, the reference position determination unit 103 moves the position of the reference point according to the direction of the arrow. The figure shows a case in which the user pinches and moves the upward arrow, in which case the user can move the position of the reference point only upward. After that, when the user releases the pinched arrow or stops it and a certain period of time has passed, the reference position determination unit 103 determines the position of the reference point at that position. At this time, the reference position determination unit 103 may display a mark as described above.

[0043] In this way, in the virtual object display device 1 according to the first embodiment, by using a bounding box, it becomes possible to easily re-determine (fine-tune) the position of the reference point. Furthermore, in the virtual object display device 1 according to the first embodiment, fine adjustment using an arrow makes it easier to fine-tune the position of the reference point. That is, for example, when it is desired to move the position of the reference point slightly straight up, it is difficult to precisely specify the position using the position of the hand, but it is easy to precisely specify the position by moving the arrow. Furthermore, for example, when an information processing device having the function of the virtual object display device 1 has a function of performing processes other than displaying virtual objects based on user gestures, the above-described fine-tuning using the bounding box is considered to effectively function as a prevention of malfunction. That is, when the information processing device performs multiple processes based on user gestures, it is necessary to prevent a gesture performed by the user for a process other than displaying a virtual object from being mistakenly recognized as a gesture for displaying a virtual object. Therefore, by employing such fine-tuning using the bounding box, it is possible to differentiate the gesture from a gesture for a process other than displaying a virtual object, which can contribute to the prevention of malfunction.

[0044] In conventional technology, real space and virtual space are aligned using a single marker such as a QR code as a reference, which can lead to significant positional deviations depending on the marker's position or angle, and also takes time to adjust the alignment. In contrast, in virtual object display device 1 according to embodiment 1, a single position adjustment and angle adjustment are performed by a gesture operation, thereby aligning the real space with the virtual space. More specifically, in virtual object display device 1 according to embodiment 1, the position of a single reference point and the angle of a reference line in the real space are determined based on the gesture detected by gesture detection unit 102, and the virtual object is displayed after aligning the position of the single reference point, the angle of the reference line, and the detected horizontal direction. As a result, in the virtual object display device 1 according to the first embodiment, by setting a plurality of parameters, the accuracy of alignment is increased compared to conventional methods. Furthermore, in the virtual object display device 1 according to the first embodiment, it is not necessary to prepare markers in advance, and the time required to complete alignment can be reduced. Furthermore, in the virtual object display device 1 according to the first embodiment, detection in the horizontal direction is performed using a sensor capable of spatial recognition, thereby improving the accuracy of alignment compared to alignment using markers.

[0045] Furthermore, the virtual object display device 1 according to the first embodiment can display a reference line that extends far away from the reference point, making it possible to align one reference point with a location far away from that reference point. This allows the virtual object display device 1 according to the first embodiment to further reduce errors.

[0046] The virtual object display device 1 according to embodiment 1 can achieve the above-described effects, and is therefore considered to be particularly effective when the virtual object is a large virtual object or a virtual object that is a combination of multiple objects, where even a small deviation can result in a large deviation.

[0047] Furthermore, in the virtual object display device 1 according to the first embodiment, by using a method of performing alignment by one position adjustment and angle adjustment, the user can perform alignment with a smaller amount of movement. That is, in the virtual object display device 1 according to the first embodiment, it is possible to display a reference line that extends far away from a reference point, and even when performing alignment between one reference point and a location far away from the reference point, the user can perform alignment while looking at the reference line, so that alignment can be easily performed without moving far away from the position of the reference point.

[0048] Furthermore, for example, the gesture detection unit 102 according to the first embodiment may detect a gesture made by a hand. This allows the user to easily perform positioning by, for example, aligning a finger with the corner of a specific object existing in real space. Furthermore, for example, when adjusting the height to the floor, the user can perform height adjustment by simply touching a corresponding part of the floor with their hand. Furthermore, for example, the gesture detection unit 102 according to the first embodiment may detect a gesture made with a foot. This makes it possible to more easily adjust the height, for example, when making a gesture with a hand to match the height of the floor, since bending down is required when making a gesture with a foot, but bending down is not required when making a gesture with a foot. Furthermore, for example, the gesture detection unit 102 according to the first embodiment may detect gestures made with hands and feet. This allows the user to more easily perform height adjustment in the height direction by making gestures with hands or feet depending on the height at which the positioning is to be performed. Furthermore, for example, the gesture detection unit 102 according to the first embodiment may detect a gesture based on eye gaze. However, since it is expected that it is difficult to focus the gaze on a specific location and keep the gaze stationary at that location for a certain period of time, it is desirable that the virtual object display device 1 have a function to correct eye gaze blur.

[0049] As described above, according to the first embodiment, the virtual object display device 1 includes a gesture detection unit 102 that detects a gesture, a reference position determination unit 103 that determines the position of a reference point in real space based on the detection result by the gesture detection unit 102, and a virtual object display unit 106 that aligns a virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit 103 and then displays the virtual object. Furthermore, according to this embodiment 1, the reference position determination unit 103 determines the position of one reference point in real space based on the detection result by the gesture detection unit 102, and is provided with a reference angle determination unit 104 that determines the angle of a reference line in real space based on the detection result by the gesture detection unit 102, and a horizontal direction detection unit 105 that detects the horizontal direction in real space, and the virtual object display unit 106 aligns a virtual object with respect to real space and displays it based on the position of the one reference point determined by the reference position determination unit 103, the angle of the reference line determined by the reference angle determination unit 104, and the horizontal direction detected by the horizontal direction detection unit 105. As a result, the virtual object display device 1 according to the first embodiment can display a virtual object after aligning it with the real space without using a marker.

[0050] Furthermore, according to this embodiment 1, the reference position determination unit 103 displays a mark at the gesture position in real space when the gesture is stationary, and when the gesture remains stationary after a certain period of time has elapsed, determines the gesture position as the position of the reference point. Furthermore, according to the first embodiment, the reference position determining unit 103 reduces the size of the mark over time so that the mark disappears after a certain period of time has passed. As a result, the virtual object display device 1 according to the first embodiment allows the user to more easily determine the position of the reference point.

[0051] Furthermore, according to this embodiment 1, the reference position determination unit 103 displays a bounding box at the position of the reference point in real space, and moves the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit 102. Furthermore, according to this embodiment 1, when the gesture position detected by the gesture detection unit 102 is located inside the bounding box, the reference position determination unit 103 cancels the determination of the position of the reference point and enables it to be re-determined. Furthermore, according to this embodiment 1, the bounding box has an arrow indicating the movement direction, and the reference position determination unit 103 moves the position of the reference point in the movement direction indicated by the arrow in accordance with a gesture made with respect to the arrow in the bounding box detected by the gesture detection unit 102. As a result, in the virtual object display device 1 according to the first embodiment, the user can more easily re-determine (fine-tune) the position of the reference point.

[0052] Furthermore, according to this embodiment 1, when the gesture is stationary, the reference angle determination unit 104 displays a reference line including the gesture position in real space and the position of the reference point determined by the reference position determination unit 103, and when the gesture remains stationary after a certain period of time has elapsed, determines the angle of the reference line. Furthermore, according to this embodiment 1, when the gesture is stationary, the reference angle determination unit 104 displays two parallel lines sandwiching a reference line, and narrows the distance between the two parallel lines over time so that the two parallel lines overlap the reference line after a certain period of time has passed. As a result, in the virtual object display device 1 according to the first embodiment, the user can more easily determine the angle of the reference line.

[0053] Furthermore, according to the first embodiment, the gesture detection unit 102 detects a gesture made by a hand. This allows the user to more easily perform positioning in the virtual object display device 1 according to the first embodiment.

[0054] Furthermore, according to the first embodiment, the virtual object display program causes the computer to function as the virtual object display device 1. As a result, the virtual object display program according to the first embodiment can display a virtual object after aligning it with the real space without using a marker.

[0055] Furthermore, according to this embodiment 1, the virtual object display method includes a step in which the gesture detection unit 102 detects a gesture; a step in which the reference position determination unit 103 determines the position of a reference point in real space based on the detection result by the gesture detection unit 102; and a step in which the virtual object display unit 106 aligns the virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit 103, and then displays the virtual object. As a result, the virtual object display method according to the first embodiment makes it possible to display a virtual object after aligning it with the real space without using a marker.

[0056] Embodiment 2 FIG. 7 is a diagram showing an example of the configuration of a virtual object display device 1b according to the second embodiment. The virtual object display device 1b displays a virtual object superimposed on a real space. This virtual object display device 1b is applicable to both AR and MR. As shown in FIG. 7, the virtual object display device 1b according to the second embodiment includes a storage unit 101b, a gesture detection unit 102b, a reference position determination unit 103b, a horizontal direction detection unit 105b, and a virtual object display unit 106b.

[0057] The storage unit 101b stores various data handled by the virtual object display device 1b. For example, storage unit 101b stores data indicating a virtual object. In the second embodiment, the positions of two reference points and a horizontal direction are set for the virtual object. Furthermore, for example, the storage unit 101b stores data indicating the positions of two reference points in real space determined by the reference position determination unit 103b. Furthermore, for example, the storage unit 101b stores data indicating the horizontal direction in real space detected by the horizontal direction detection unit 105b.

[0058] Here, the storage unit 101b may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, or EEPROM, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0059] 7 shows a case where the storage unit 101b is provided inside the virtual object display device 1b. However, the present invention is not limited to this, and the storage unit 101b may be provided outside the virtual object display device 1b.

[0060] The gesture detection unit 102b detects a gesture. In the following, an example will be described in which the gesture detection unit 102b mainly detects gestures made with the hands. However, this is not limiting, and the gesture detection unit 102b may also detect gestures made with other parts of the body, such as the feet or gaze.

[0061] The reference position determination unit 103b determines the positions of reference points in real space based on the detection result by the gesture detection unit 102b. The reference position determination unit 103b in the second embodiment determines the positions of two reference points in real space based on the detection result by the gesture detection unit 102b. The positions of the reference points determined by the reference position determination unit 103b are positions to be aligned with the positions of reference points set for the virtual object.

[0062] For example, when the gesture becomes stationary, the reference position determination unit 103b displays a mark at the gesture position in real space, and when the gesture remains stationary (remains within a predetermined range) after a certain period of time has passed, the reference position determination unit 103b determines the gesture position as the position of the reference point. At this time, the reference position determining unit 103b may reduce the size of the mark over time so that the mark disappears after a certain period of time has passed.

[0063] The user knows in advance which parts of the virtual object the positions of the two reference points determined by the reference position determination unit 103b will be aligned with, and also knows in advance the order in which the positions of the two reference points will be determined.

[0064] Furthermore, the reference position determining unit 103b may be configured to be able to re-determine the position of the reference point after the position of the reference point has been determined based on the detection result by the gesture detection unit 102b. In this case, the reference position determination unit 103b may display a bounding box at the position of the reference point in real space, and move the position of the reference point in accordance with the gesture on the bounding box detected by the gesture detection unit 102b. Note that, for example, the center position of the bounding box is aligned with the position of the reference point determined by the reference position determination unit 103b.

[0065] For example, when the gesture position detected by the gesture detection unit 102b is located inside the bounding box, the reference position determination unit 103b may cancel the determination of the position of the reference point and allow it to be determined again. Furthermore, for example, the bounding box may have an arrow indicating the direction of movement, and the reference position determination unit 103b may move the position of the reference point in the direction of movement indicated by the arrow in accordance with a gesture (for example, an action of pinching the arrow with a hand and moving it) made with respect to the arrow in the bounding box detected by the gesture detection unit 102b. In the above two methods, when the position of the first reference point is adjusted, the position of the second reference point also follows, so the positions of the first and second reference points are redefined simultaneously. Also, when the position of the second reference point is adjusted, only the position of the second reference point moves (revolves) around the position of the first reference point as the origin, so only the position of the second reference point is redefined.

[0066] The horizontal direction detection unit 105b detects the horizontal direction in real space. The horizontal direction detected by the horizontal direction detection unit 105b is a direction to be matched with the horizontal direction set for the virtual object. As the horizontal direction detection unit 105b, a sensor capable of recognizing space, which is generally mounted on the virtual object display device 1b, can be used.

[0067] Virtual object display unit 106b aligns the position of a virtual object with respect to real space based on the positions of the reference points determined by reference position determination unit 103b, and then displays the virtual object. Virtual object display unit 106b in the second embodiment aligns the position of a virtual object with respect to real space based on the positions of the two reference points determined by reference position determination unit 103b and the horizontal direction detected by horizontal direction detection unit 105b, and then displays the virtual object. That is, the virtual object display unit 106b aligns the positions of the two reference points and the horizontal direction set for the virtual object stored in the memory unit 101b with the positions of the two reference points determined by the reference position determination unit 103b and the horizontal direction detected by the horizontal direction detection unit 105b, and then displays the virtual object in real space.

[0068] Next, an example of the operation of the virtual object display device 1b according to the second embodiment shown in Fig. 7 will be described with reference to Fig. 8. Note that the gesture detection unit 102b constantly detects gestures. In the operation example of the virtual object display device 1b according to the second embodiment shown in FIG. 7, as shown in FIG. 8, for example, the reference position determination unit 103b first determines the positions of two reference points in the real space based on the detection result by the gesture detection unit 102b (step ST201).

[0069] For example, when the gesture becomes stationary, the reference position determination unit 103b displays a mark at the gesture position in real space, and when the gesture remains stationary (remains within a predetermined range) after a certain period of time has passed, the reference position determination unit 103b determines the gesture position as the position of the reference point. At this time, the reference position determining unit 103b may reduce the size of the mark over time so that the mark disappears after a certain period of time has passed.

[0070] Furthermore, the horizontal direction detection unit 105b detects the horizontal direction in the real space (step ST202).

[0071] Next, the virtual object display unit 106b aligns the virtual object with the real space based on the positions of the two reference points determined by the reference position determination unit 103b and the horizontal direction detected by the horizontal direction detection unit 105b, and then displays the virtual object (step ST203). That is, the virtual object display unit 106b aligns the positions of the two reference points and the horizontal direction set for the virtual object stored in the memory unit 101b with the positions of the two reference points determined by the reference position determination unit 103b and the horizontal direction detected by the horizontal direction detection unit 105b, and then displays the virtual object in real space.

[0072] Furthermore, the reference position determining unit 103b may be configured to be able to re-determine the position of the reference point after the position of the reference point has been determined based on the detection result by the gesture detection unit 102b. In this case, the reference position determination unit 103b may display a bounding box at the position of the reference point in real space, and move the position of the reference point in accordance with the gesture on the bounding box detected by the gesture detection unit 102b. Note that, for example, the center position of the bounding box is aligned with the position of the reference point determined by the reference position determination unit 103b.

[0073] For example, when the gesture position detected by the gesture detection unit 102b is located inside the bounding box, the reference position determination unit 103b may cancel the determination of the position of the reference point and allow it to be determined again. Furthermore, for example, the bounding box may have an arrow indicating the direction of movement, and the reference position determination unit 103b may move the position of the reference point in the direction of movement indicated by the arrow in accordance with a gesture (for example, an action of pinching the arrow with a hand and moving it) made with respect to the arrow in the bounding box detected by the gesture detection unit 102b. In the above two methods, when the position of the first reference point is adjusted, the position of the second reference point also follows, so the positions of the first and second reference points are redefined simultaneously. Also, when the position of the second reference point is adjusted, only the position of the second reference point moves (revolves) around the position of the first reference point as the origin, so only the position of the second reference point is redefined.

[0074] Next, an example of the operation of the virtual object display device 1b according to the second embodiment will be described with reference to Fig. 9. In the following example, it is assumed that the gesture detection unit 102b detects a gesture made by a hand.

[0075] In this case, first, as shown in Fig. 9A, the reference position determination unit 103b determines the position of a first reference point in real space based on the hand gesture detected by the gesture detection unit 102b. In the example of Fig. 9A, the hand is placed on the corner of a foreground object in real space. The specific operation performed by the reference position determining unit 103b at this time is the same as the operation shown in Fig. 4. In the figure, reference numeral 91 denotes a mark displayed at the gesture position.

[0076] 9, the virtual object is displayed before the position of the first reference point is determined by the reference position determination unit 103b. In this case, the virtual object display unit 106b displays the virtual object according to the position of the first reference point before determination and the position of the provisional second reference point. Furthermore, when the position of the first reference point before determination moves based on the detection result by the gesture detection unit 102b, the virtual object display unit 106b changes the position of the virtual object accordingly. In this way, in the virtual object display device 1b according to the second embodiment, the virtual object is displayed before the reference position determination unit 103b determines the position of the first reference point, so that the user can perform the operation of determining the position of the reference point while checking the display state of the virtual object, and it becomes easier to determine the position of the reference point.

[0077] Then, as shown in Fig. 9B, the reference position determination unit 103b determines the position of the second reference point in real space based on the hand gesture detected by the gesture detection unit 102b. In the example of Fig. 9B, the hand is placed on the corner of an object at the back in real space. The specific operation performed by the reference position determining unit 103b at this time is the same as the operation shown in Fig. 4. In the figure, reference numeral 92 denotes a mark displayed at the gesture position.

[0078] 9, the virtual object is displayed before the position of the second reference point is determined by the reference position determination unit 103b. In this case, the virtual object display unit 106b displays the virtual object according to the position of the first reference point after determination and the position of the second reference point before determination. Furthermore, when the position of the second reference point before determination moves based on the detection result by the gesture detection unit 102b, the virtual object display unit 106b changes the position of the virtual object accordingly. In this way, in the virtual object display device 1b according to the second embodiment, the virtual object is displayed before the position of the second reference point is determined by the reference position determination unit 103b, so that the user can perform the operation of determining the position of the reference point while checking the display state of the virtual object, and it becomes easier to determine the position of the reference point.

[0079] In conventional technology, real space and virtual space are aligned using a single marker such as a QR code as a reference, which can lead to significant positional deviations depending on the marker's position or angle, and also takes time to adjust the alignment. In contrast, in virtual object display device 1b according to embodiment 2, the real space and the virtual space are aligned by performing two position adjustments using gesture operations. More specifically, in virtual object display device 1b according to embodiment 2, the positions of two reference points in the real space are determined based on the gesture detected by gesture detection unit 102b, and the virtual object is displayed after aligning the positions of the two reference points and the detected horizontal direction. As a result, in virtual object display device 1b according to embodiment 2, by setting a plurality of parameters, the accuracy of alignment is increased compared to conventional methods. Also, in virtual object display device 1b according to embodiment 2, it is not necessary to prepare markers in advance, and the time required to complete alignment can be reduced. Furthermore, in the virtual object display device 1b according to the second embodiment, the horizontal direction is detected using a sensor capable of spatial recognition, thereby improving the accuracy of alignment compared to alignment using markers.

[0080] Furthermore, in virtual object display device 1b according to embodiment 1, two reference points can be set apart, and positioning can be performed using two reference points that are apart from each other. This allows virtual object display device 1b according to embodiment 1 to further reduce errors.

[0081] The virtual object display device 1b according to the second embodiment can achieve the above-described effects, and is therefore considered to be particularly effective when the virtual object is a large virtual object or a virtual object that is a combination of multiple objects, where even a small deviation can result in a large deviation.

[0082] Furthermore, in virtual object display device 1b according to embodiment 2, by adopting a method of performing alignment through two position adjustments, it is believed that alignment can be performed more easily than in embodiment 1. That is, while virtual object display device 1 according to embodiment 1 requires the work of adjusting the angle of the reference line, virtual object display device 1b according to embodiment 2 does not require such work and only requires specifying the positions of two reference points, making alignment easier.

[0083] Furthermore, for example, the gesture detection unit 102b according to the second embodiment may detect a gesture made by a hand. This allows the user to easily perform positioning by, for example, aligning a finger with the corner of a specific object existing in real space. Furthermore, for example, when adjusting the height to the floor, the user can perform height adjustment by simply touching a corresponding part of the floor with their hand. Furthermore, for example, the gesture detection unit 102b according to the second embodiment may detect a gesture made with a foot. This makes it possible to more easily adjust the height, for example, when making a gesture with a hand, such as when adjusting the height to the floor, since bending down is required when making a gesture with a foot, but bending down is not required when making a gesture with a foot. Furthermore, for example, the gesture detection unit 102b according to the second embodiment may detect gestures made with hands and feet. This allows the user to more easily perform height adjustment in the height direction by making gestures with hands or feet depending on the height at which the adjustment is to be performed. Furthermore, for example, the gesture detection unit 102b according to the second embodiment may detect a gesture based on eye gaze. However, since it is expected that it is difficult to focus the gaze on a specific location and keep the gaze stationary at that location for a certain period of time, it is desirable that the virtual object display device 1b have a function to correct eye gaze blur.

[0084] As described above, according to the second embodiment, the reference position determination unit 103b determines the positions of two reference points in real space based on the detection result by the gesture detection unit 102b, and includes a horizontal direction detection unit 105b that detects the horizontal direction in real space, and the virtual object display unit 106b aligns the virtual object with the real space and displays it based on the positions of the two reference points determined by the reference position determination unit 103b and the horizontal direction detected by the horizontal direction detection unit 105b. As a result, the virtual object display device 1b according to the second embodiment can display a virtual object after aligning it with the real space without using a marker.

[0085] Embodiment 3 FIG. 10 is a diagram showing an example of the configuration of a virtual object display device 1c according to the third embodiment. The virtual object display device 1c displays a virtual object superimposed on a real space. This virtual object display device 1c is applicable to both AR and MR. As shown in FIG. 10 , the virtual object display device 1c according to the third embodiment includes a storage unit 101c, a gesture detection unit 102c, a reference position determination unit 103c, and a virtual object display unit 106c.

[0086] The storage unit 101c stores various data handled by the virtual object display device 1c. For example, the storage unit 101c stores data indicating a virtual object. In the third embodiment, the positions of three reference points are set for the virtual object. Furthermore, for example, the storage unit 101c stores data indicating the positions of three reference points in real space determined by the reference position determination unit 103c.

[0087] Here, the storage unit 101c may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, or EEPROM, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0088] 10 shows a case where the storage unit 101c is provided inside the virtual object display device 1c, but is not limited to this, and the storage unit 101c may be provided outside the virtual object display device 1c.

[0089] The gesture detection unit 102c detects a gesture. In the following, an example will be described in which the gesture detection unit 102c mainly detects gestures made with the hands. However, this is not limiting, and the gesture detection unit 102c may also detect gestures made with other parts such as the feet or gaze.

[0090] The reference position determination unit 103c determines the positions of reference points in real space based on the detection result by the gesture detection unit 102c. The reference position determination unit 103c in the third embodiment determines the positions of three reference points in real space based on the detection result by the gesture detection unit 102c. The positions of the reference points determined by the reference position determination unit 103c are positions to be aligned with the positions of reference points set for the virtual object.

[0091] For example, the reference position determination unit 103c displays a mark at the gesture position in real space when the gesture is stationary, and if the gesture remains stationary (remains within a predetermined range) after a certain period of time has passed, determines the gesture position as the position of the reference point. At this time, the reference position determining unit 103c may reduce the size of the mark over time so that the mark disappears after a certain period of time has passed.

[0092] The user knows in advance which parts of the virtual object the positions of the three reference points determined by the reference position determination unit 103c will be aligned with, and also knows in advance the order in which the positions of the three reference points will be determined.

[0093] Furthermore, the reference position determining unit 103c may be configured to be able to re-determine the position of the reference point after the position of the reference point has been determined based on the detection result by the gesture detection unit 102c. In this case, the reference position determination unit 103c may display a bounding box at the position of the reference point in real space and move the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit 102c. Note that, for example, the center position of the bounding box is aligned with the position of the reference point determined by the reference position determination unit 103c.

[0094] For example, when the gesture position detected by the gesture detection unit 102c is located inside the bounding box, the reference position determination unit 103c may cancel the determination of the position of the reference point and allow it to be determined again. Also, for example, the bounding box may have an arrow indicating the direction of movement, and the reference position determination unit 103c may move the position of the reference point in the direction of movement indicated by the arrow in accordance with a gesture (for example, an action of pinching the arrow with a hand and moving it) made with respect to the arrow in the bounding box detected by the gesture detection unit 102c. In the two methods above, when the position of the first reference point is adjusted, the positions of the second and third reference points also follow, so the positions of the first, second, and third reference points are redefined simultaneously. Also, when the position of the second reference point is adjusted, the position of the second reference point moves using the position of the first reference point as the origin, and the position of the third reference point follows, so the positions of the second and third reference points are redefined simultaneously. Also, when the position of the third reference point is adjusted, only the position of the third reference point moves using the line connecting the positions of the first and second reference points as the axis, so only the position of the third reference point is redefined.

[0095] The virtual object display unit 106c aligns the position of the virtual object with respect to real space based on the positions of the reference points determined by the reference position determination unit 103c, and then displays the virtual object. The virtual object display unit 106c in the third embodiment aligns the position of the virtual object with respect to real space based on the positions of the three reference points determined by the reference position determination unit 103c, and then displays the virtual object. That is, the virtual object display unit 106c aligns the positions of the three reference points set for the virtual object stored in the memory unit 101c with the positions of the three reference points determined by the reference position determination unit 103c, and then displays the virtual object in real space.

[0096] Next, an example of the operation of the virtual object display device 1c according to the third embodiment shown in Fig. 10 will be described with reference to Fig. 11. Note that the gesture detection unit 102c constantly detects gestures. In the operation example of the virtual object display device 1c according to the third embodiment shown in FIG. 10, as shown in FIG. 11, for example, the reference position determination unit 103c first determines the positions of three reference points in the real space based on the detection result by the gesture detection unit 102c (step ST301).

[0097] For example, the reference position determination unit 103c displays a mark at the gesture position in real space when the gesture is stationary, and if the gesture remains stationary (remains within a predetermined range) after a certain period of time has passed, determines the gesture position as the position of the reference point. At this time, the reference position determining unit 103c may reduce the size of the mark over time so that the mark disappears after a certain period of time has passed.

[0098] Next, the virtual object display unit 106c aligns the virtual object with the real space based on the positions of the three reference points determined by the reference position determining unit 103c, and then displays the virtual object (step ST302). That is, the virtual object display unit 106c aligns the positions of the three reference points set for the virtual object stored in the memory unit 101c with the positions of the three reference points determined by the reference position determination unit 103c, and then displays the virtual object in real space.

[0099] Furthermore, the reference position determining unit 103c may be configured to be able to re-determine the position of the reference point after the position of the reference point has been determined based on the detection result by the gesture detection unit 102c. In this case, the reference position determination unit 103c may display a bounding box at the position of the reference point in real space and move the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit 102c. Note that, for example, the center position of the bounding box is aligned with the position of the reference point determined by the reference position determination unit 103c.

[0100] For example, when the gesture position detected by the gesture detection unit 102c is located inside the bounding box, the reference position determination unit 103c may cancel the determination of the position of the reference point and allow it to be determined again. Also, for example, the bounding box may have an arrow indicating the direction of movement, and the reference position determination unit 103c may move the position of the reference point in the direction of movement indicated by the arrow in accordance with a gesture (for example, an action of pinching the arrow with a hand and moving it) made with respect to the arrow in the bounding box detected by the gesture detection unit 102c. In the two methods above, when the position of the first reference point is adjusted, the positions of the second and third reference points also follow, so the positions of the first, second, and third reference points are redefined simultaneously. Also, when the position of the second reference point is adjusted, the position of the second reference point moves using the position of the first reference point as the origin, and the position of the third reference point follows, so the positions of the second and third reference points are redefined simultaneously. Also, when the position of the third reference point is adjusted, only the position of the third reference point moves using the line connecting the positions of the first and second reference points as the axis, so only the position of the third reference point is redefined.

[0101] Next, an example of the operation of the virtual object display device 1c according to the third embodiment will be described with reference to Fig. 12. In the following example, it is assumed that the gesture detection unit 102c detects a gesture made by a hand.

[0102] In this case, first, as shown in Fig. 12A, the reference position determination unit 103c determines the position of a first reference point in real space based on the hand gesture detected by the gesture detection unit 102c. In the example of Fig. 12A, the hand is placed on the corner of a foreground object existing in real space. The specific operation of the reference position determining unit 103c at this time is the same as the operation shown in Fig. 4. In the figure, reference numeral 121 denotes a mark displayed at the gesture position.

[0103] 12, the virtual object is displayed before the position of the first reference point is determined by the reference position determination unit 103c. In this case, the virtual object display unit 106c displays the virtual object according to the position of the first reference point before determination, the position of the provisional second reference point, and the position of the provisional third reference point. Furthermore, when the position of the first reference point before determination moves based on the detection result by the gesture detection unit 102c, the virtual object display unit 106c changes the position of the virtual object accordingly. In this way, in the virtual object display device 1c according to the third embodiment, the virtual object is displayed before the reference position determination unit 103c determines the position of the second reference point, so that the user can determine the position of the reference point while checking the display state of the virtual object, thereby making it easier to determine the position of the reference point.

[0104] 12B, the reference position determination unit 103c determines the position of the second reference point in the real space based on the hand gesture detected by the gesture detection unit 102c. In the example of FIG. 12A, the hand is placed on the front corner of an object in the back that exists in the real space. The specific operation of the reference position determining unit 103c at this time is the same as the operation shown in Fig. 4. In the figure, reference numeral 122 denotes a mark displayed at the gesture position.

[0105] 9, the virtual object is displayed before the position of the second reference point is determined by the reference position determination unit 103c. In this case, the virtual object display unit 106c displays the virtual object according to the position of the first reference point after determination, the position of the second reference point before determination, and the position of the provisional third reference point. Furthermore, when the position of the second reference point before determination moves based on the detection result by the gesture detection unit 102c, the virtual object display unit 106c changes the position of the virtual object accordingly. In this way, in the virtual object display device 1c according to the third embodiment, the virtual object is displayed before the reference position determination unit 103c determines the position of the second reference point, so that the user can determine the position of the reference point while checking the display state of the virtual object, thereby making it easier to determine the position of the reference point.

[0106] 12C, the reference position determination unit 103c determines the position of the third reference point in the real space based on the hand gesture detected by the gesture detection unit 102c. In the example of FIG. 12A, the hand is placed on the far corner of the far object existing in the real space. The specific operation of the reference position determining unit 103c at this time is the same as the operation shown in Fig. 4. In the figure, reference numeral 123 denotes a mark displayed at the gesture position.

[0107] 9, the virtual object is displayed before the position of the third reference point is determined by the reference position determination unit 103c. In this case, the virtual object display unit 106c displays the virtual object according to the position of the first reference point after determination, the position of the second reference point after determination, and the position of the third reference point before determination. Furthermore, when the position of the third reference point before determination moves based on the detection result by the gesture detection unit 102c, the virtual object display unit 106c changes the position of the virtual object accordingly. In this way, in the virtual object display device 1c according to the third embodiment, the virtual object is displayed before the reference position determination unit 103c determines the position of the third reference point, so that the user can determine the position of the reference point while checking the display state of the virtual object, making it easier to determine the position of the reference point.

[0108] In conventional technology, real space and virtual space are aligned using a single marker such as a QR code as a reference, which can lead to significant positional deviations depending on the marker's position or angle, and also takes time to adjust the alignment. In contrast, in virtual object display device 1c according to embodiment 3, three position adjustments are performed by gesture operations to align the real space with the virtual space. More specifically, in virtual object display device 1c according to embodiment 3, the positions of three reference points in the real space are determined based on the gesture detected by gesture detection unit 102c, and the virtual object is displayed after aligning the position at the positions of the three reference points. As a result, in the virtual object display device 1c according to the third embodiment, by setting a plurality of parameters, the accuracy of alignment is increased compared to conventional methods. Furthermore, in the virtual object display device 1c according to the third embodiment, it is not necessary to prepare markers in advance, and the time required to complete alignment can be reduced.

[0109] Furthermore, the virtual object display device 1c according to the third embodiment does not require horizontal detection as in the first and second embodiments, and therefore can be configured more simply. Furthermore, in the virtual object display device 1c according to the third embodiment, horizontal direction detection may be performed and this information may also be used for positioning, which is thought to further improve accuracy.

[0110] Furthermore, in virtual object display device 1c according to embodiment 3, three reference points can be set at a distance from each other, and positioning can be performed using three reference points that are at a distance from each other. This allows virtual object display device 1c according to embodiment 3 to further reduce errors.

[0111] The virtual object display device 1c according to the third embodiment can achieve the above-described effects, and is therefore considered to be particularly effective when the virtual object is a large virtual object or a virtual object that is a combination of multiple objects, where even a small deviation can result in a large deviation.

[0112] Furthermore, in virtual object display device 1c according to embodiment 3, by adopting a method of performing alignment through three position adjustments, it is believed that alignment can be performed more easily than in embodiment 1. That is, while virtual object display device 1 according to embodiment 1 requires the work of adjusting the angle of the reference line, virtual object display device 1c according to embodiment 3 does not require such work and only requires specifying the positions of three reference points, making alignment easier.

[0113] Furthermore, for example, the gesture detection unit 102c according to the third embodiment may detect a gesture made by a hand. This allows the user to easily perform positioning by, for example, aligning a finger with the corner of a specific object existing in real space. Furthermore, for example, when adjusting the height to the floor, the user can perform height adjustment by simply touching a corresponding part of the floor with their hand. Furthermore, for example, the gesture detection unit 102c according to the third embodiment may detect a gesture made with a foot. This makes it possible to more easily adjust the height, for example, when making a gesture with a hand, such as when adjusting the height to the floor, since bending down is required when making a gesture with a foot, but bending down is not required when making a gesture with a foot. Furthermore, for example, the gesture detection unit 102c according to the third embodiment may detect gestures made with hands and feet. This allows the user to more easily perform height adjustment in the height direction by making gestures with hands or feet depending on the height at which the adjustment is to be performed. Furthermore, for example, the gesture detection unit 102c according to the third embodiment may detect a gesture based on eye gaze. However, since it is expected that it is difficult to focus the gaze on a specific location and keep the gaze stationary at that location for a certain period of time, it is desirable that the virtual object display device 1c has a function to correct eye gaze blur.

[0114] As described above, according to the third embodiment, the reference position determination unit 103c determines the positions of three reference points in the real space based on the detection results by the gesture detection unit 102c, and the virtual object display unit 106c aligns the virtual object with the real space based on the positions of the three reference points determined by the reference position determination unit 103c, and then displays the virtual object. As a result, the virtual object display device 1c according to the third embodiment can display a virtual object after aligning it with the real space without using a marker.

[0115] 13, examples of the hardware configuration of the virtual object display device 1 according to the first embodiment, the virtual object display device 1b according to the second embodiment, and the virtual object display device 1c according to the third embodiment will be described. Below, an example of the hardware configuration of the virtual object display device 1 according to the first embodiment will be described, but the same applies to the example of the hardware configuration of the virtual object display device 1b according to the second embodiment and the virtual object display device 1c according to the third embodiment. The functions of the gesture detection unit 102, the reference position determination unit 103, the reference angle determination unit 104, the horizontal direction detection unit 105, and the virtual object display unit 106 in the virtual object display device 1 are realized by a processing circuit 501. The processing circuit 501 may be dedicated hardware as shown in Fig. 13A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 502 that executes a program stored in a memory 503 as shown in Fig. 13B.

[0116] When the processing circuit 501 is dedicated hardware, the processing circuit 501 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the gesture detection unit 102, the reference position determination unit 103, the reference angle determination unit 104, the horizontal direction detection unit 105, and the virtual object display unit 106 may be realized individually by the processing circuit 501, or the functions of each unit may be realized collectively by the processing circuit 501.

[0117] When the processing circuit 501 is a CPU 502, the functions of the gesture detection unit 102, the reference position determination unit 103, the reference angle determination unit 104, the horizontal direction detection unit 105, and the virtual object display unit 106 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 503. The processing circuit 501 realizes the functions of each unit by reading and executing the programs stored in the memory 503. That is, the virtual object display device 1 includes the memory 503 for storing a program that, when executed by the processing circuit 501, results in the execution of, for example, each step shown in FIG. 2 . It can also be said that these programs cause a computer to execute the procedures and methods of the gesture detection unit 102, the reference position determination unit 103, the reference angle determination unit 104, the horizontal direction detection unit 105, and the virtual object display unit 106. Here, memory 503 may be, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, or EEPROM, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0118] Note that the functions of the gesture detection unit 102, the reference position determination unit 103, the reference angle determination unit 104, the horizontal direction detection unit 105, and the virtual object display unit 106 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the function of the gesture detection unit 102 may be realized by a processing circuit 501 as dedicated hardware, and the functions of the reference position determination unit 103, the reference angle determination unit 104, the horizontal direction detection unit 105, and the virtual object display unit 106 may be realized by the processing circuit 501 reading and executing programs stored in a memory 503.

[0119] In this way, the processing circuitry 501 can realize each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0120] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0121] Various aspects of the present disclosure are summarized below as appendices.

[0122] (Appendix 1) a gesture detection unit that detects a gesture; a reference position determination unit that determines the position of a reference point in real space based on a detection result by the gesture detection unit; a virtual object display unit that aligns a virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit and displays the virtual object; A virtual object display device comprising: (Appendix 2) the reference position determination unit determines the position of one reference point in real space based on a detection result by the gesture detection unit; a reference angle determination unit that determines an angle of a reference line in real space based on a detection result by the gesture detection unit; a horizontal direction detection unit that detects a horizontal direction in real space, The virtual object display unit aligns a virtual object with respect to real space based on the position of one reference point determined by the reference position determination unit, the angle of the reference line determined by the reference angle determination unit, and the horizontal direction detected by the horizontal direction detection unit, and then displays the virtual object. 2. A virtual object display device according to claim 1. (Appendix 3) the reference position determination unit determines positions of two reference points in real space based on a detection result by the gesture detection unit; a horizontal direction detection unit that detects a horizontal direction in real space; The virtual object display unit aligns the position of the virtual object with respect to the real space based on the positions of the two reference points determined by the reference position determination unit and the horizontal direction detected by the horizontal direction detection unit, and then displays the virtual object. 2. A virtual object display device according to claim 1. (Appendix 4) the reference position determination unit determines positions of three reference points in real space based on a detection result by the gesture detection unit; The virtual object display unit aligns the virtual object with the real space based on the positions of the three reference points determined by the reference position determination unit, and then displays the virtual object. 2. A virtual object display device according to claim 1. (Appendix 5) The reference position determination unit displays a mark at the gesture position in real space when the gesture stops, and determines the gesture position as the position of the reference point when the gesture remains stationary after a certain period of time has elapsed. 5. The virtual object display device according to claim 1, wherein the virtual object display device is a display device for displaying a virtual object on a display screen. (Appendix 6) The reference position determination unit reduces the mark over time so that the mark disappears after the predetermined time has elapsed. 6. A virtual object display device according to claim 5. (Appendix 7) The reference position determination unit displays a bounding box at the position of the reference point in real space, and moves the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit. 7. A virtual object display device according to claim 1, wherein the virtual object display device is a display device for displaying a virtual object on a display screen. (Appendix 8) The reference position determination unit cancels the determination of the position of the reference point and enables re-determination when the gesture position detected by the gesture detection unit is located inside the bounding box. 8. The virtual object display device according to claim 7. (Appendix 9) the bounding box has an arrow indicating the direction of movement; The reference position determination unit moves the position of the reference point in a movement direction indicated by the arrow in accordance with a gesture made with respect to the arrow in the bounding box detected by the gesture detection unit. 9. The virtual object display device according to claim 7 or 8. (Appendix 10) The reference angle determination unit displays a reference line including the gesture position in real space and the position of the reference point determined by the reference position determination unit when the gesture is stationary, and determines the angle of the reference line when the gesture remains stationary after a certain period of time has elapsed. 3. The virtual object display device according to claim 2. (Appendix 11) The reference angle determination unit displays two parallel lines sandwiching the reference line when the gesture is stationary, and narrows the gap between the two parallel lines over time so that the two parallel lines overlap the reference line when the certain time has elapsed. 11. The virtual object display device according to claim 10. (Appendix 12) The gesture detection unit detects a gesture made by a hand. 12. The virtual object display device according to claim 1, wherein the virtual object display device is a display device for displaying a virtual object. (Appendix 13) A virtual object display program for causing a computer to function as a virtual object display device according to any one of Supplementary Note 1 to Supplementary Note 12. (Appendix 14) a gesture detection unit detecting a gesture; a step in which a reference position determination unit determines a position of a reference point in real space based on a detection result by the gesture detection unit; a virtual object display unit aligning the virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit and then displaying the virtual object; A virtual object display method comprising: [Explanation of symbols]

[0123] 1, 1b, 1c virtual object display device, 101, 101b, 101c memory unit, 102, 102b, 102c gesture detection unit, 103, 103b, 103c reference position determination unit, 104 reference angle determination unit, 105, 105b horizontal direction detection unit, 106, 106b, 106c virtual object display unit, 501 processing circuit, 502 CPU, 503 memory.

Claims

1. a gesture detection unit that detects a gesture; a reference position determination unit that determines the position of a reference point in real space based on a detection result by the gesture detection unit; a virtual object display unit that aligns a virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit and displays the virtual object; A virtual object display device comprising:

2. the reference position determination unit determines the position of one reference point in real space based on a detection result by the gesture detection unit; a reference angle determination unit that determines an angle of a reference line in real space based on a detection result by the gesture detection unit; a horizontal direction detection unit that detects a horizontal direction in real space, The virtual object display unit aligns a virtual object with respect to real space based on the position of one reference point determined by the reference position determination unit, the angle of the reference line determined by the reference angle determination unit, and the horizontal direction detected by the horizontal direction detection unit, and then displays the virtual object.

2. The virtual object display device according to claim 1.

3. the reference position determination unit determines positions of two reference points in real space based on a detection result by the gesture detection unit; a horizontal direction detection unit that detects a horizontal direction in real space; The virtual object display unit aligns the position of the virtual object with respect to real space based on the positions of the two reference points determined by the reference position determination unit and the horizontal direction detected by the horizontal direction detection unit, and then displays the virtual object.

2. The virtual object display device according to claim 1.

4. the reference position determination unit determines positions of three reference points in real space based on a detection result by the gesture detection unit; The virtual object display unit aligns the virtual object with respect to real space based on the positions of the three reference points determined by the reference position determination unit, and then displays the virtual object.

2. The virtual object display device according to claim 1.

5. The reference position determination unit displays a mark at the gesture position in real space when the gesture stops, and determines the gesture position as the position of the reference point when the gesture remains stationary after a certain period of time has elapsed.

2. The virtual object display device according to claim 1.

6. The reference position determination unit reduces the mark over time so that the mark disappears after the predetermined time has elapsed.

6. The virtual object display device according to claim 5.

7. The reference position determination unit displays a bounding box at the position of the reference point in real space, and moves the position of the reference point in accordance with a gesture on the bounding box detected by the gesture detection unit.

2. The virtual object display device according to claim 1.

8. The reference position determination unit cancels the determination of the position of the reference point and enables re-determination when the gesture position detected by the gesture detection unit is located inside the bounding box.

8. The virtual object display device according to claim 7.

9. the bounding box has an arrow indicating the direction of movement; The reference position determination unit moves the position of the reference point in a movement direction indicated by the arrow in accordance with a gesture made with respect to the arrow in the bounding box detected by the gesture detection unit.

8. The virtual object display device according to claim 7.

10. The reference angle determination unit displays a reference line including the gesture position in real space and the position of the reference point determined by the reference position determination unit when the gesture is stationary, and determines the angle of the reference line when the gesture remains stationary after a certain period of time has elapsed.

3. The virtual object display device according to claim 2.

11. The reference angle determination unit displays two parallel lines sandwiching the reference line when the gesture is stationary, and narrows the gap between the two parallel lines over time so that the two parallel lines overlap the reference line when the certain time has elapsed.

11. The virtual object display device according to claim 10.

12. The gesture detection unit detects a gesture made by a hand.

2. The virtual object display device according to claim 1.

13. A virtual object display program for causing a computer to function as the virtual object display device according to claim 1.

14. a gesture detection unit detecting a gesture; a step in which a reference position determination unit determines a position of a reference point in real space based on a detection result by the gesture detection unit; a virtual object display unit aligning the virtual object with respect to real space based on the position of the reference point determined by the reference position determination unit and then displaying the virtual object; A virtual object display method comprising:

Images