Controller, system, method, and program

Abstract

To provide a controller for a transmission-type display device that can display a marker for indicating the position of an object on a transmission-type display device arranged between a viewer and the object.SOLUTION: A controller for a transmission-type display device comprises a memory, a processor, and a program stored in the memory and configured to be executed by the processor. The program comprises a first instruction for acquiring relative positions of both eyes of a viewer, an object, and the transmission-type display device disposed between the viewer and the object, a second instruction for calculating a binocular parallax of the viewer when gazing at the object from the relative positions, and a third instruction for determining the size of a marker to be displayed on the transmission-type display device in order to indicate the object.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a controller, a system, a method, and a program. [Background technology]

[0002] A display system is known in which a transmission type display device is placed between an observer and an object, and an image or the like is displayed so as to be superimposed on the object as seen by the observer (Patent Document 1). Such a display system is increasingly being used for demonstrations at trade fairs, etc., for the purpose of highlighting the product as the object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2014-503835 Summary of the Invention [Problem to be solved by the invention]

[0004] When using the above display system, if an observer gazes at an object placed in front of the transmission-type display device, the video, image, etc. displayed on the transmission-type display device may be out of focus and may be perceived as an image different from the actual object being displayed. Specifically, even if one image is displayed on the transmission-type display device, the observer may see two overlapping images due to binocular parallax.

[0005] When there are multiple objects, there is a demand to guide the observer's line of sight to each object using an image displayed on the display system. In such a case, a method of displaying (an image of) a marker indicating the object on a transmissive display device can be considered.

[0006] However, as mentioned above, simply displaying a marker on a transmissive display device may result in the observer seeing the marker twice, making it difficult to correctly point to an object. Furthermore, when using a transmissive display device with a single display surface, it is difficult to show different images or videos to each of the observer's eyes, taking binocular parallax into account. Technology for showing different images or videos to each of the observer's eyes, taking binocular parallax into account, is well known in the field of head-mounted displays, etc. However, in this case, separate display surfaces are prepared for each eye. It is difficult to provide the above technology when using a single transmissive display device with a single display surface.

[0007] The present disclosure solves at least one of the problems of the conventional techniques described above, and provides a controller for a transmissive display device that can cause a marker for indicating the position of an object to be displayed on the transmissive display device disposed between a viewer and an object. [Means for solving the problem]

[0008] The controller of the present disclosure is a controller for a transmissive display device, and includes a memory, a processor, and a program stored in the memory and configured to be executable by the processor, the program including an instruction (first instruction) to acquire the relative positions of an observer's eyes, an object, and the transmissive device positioned between the observer and the object, an instruction (second instruction) to calculate the binocular parallax of the observer when gazing at the object from the relative positions, and an instruction (third instruction) to determine the size of a marker to be displayed on the transmissive display device to indicate the object. [Effects of the Invention]

[0009] The present disclosure solves at least one of the problems of the conventional techniques described above, and provides a controller for a transmissive display device that can cause a marker for indicating the position of an object to be displayed on the transmissive display device disposed between a viewer and an object. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating a usage scenario of a system including a transmissive display device controlled by a controller according to the present disclosure. [Figure 2] FIG. 1 is a block diagram of a system including a first embodiment of a controller of the present disclosure. [Figure 3] FIG. 10 is an explanatory diagram of a procedure for calculating binocular disparity XA using an instruction set included in binocular disparity calculation module 30E. [Figure 4] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 5] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 6] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 7] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 8] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 9] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 10] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 11] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 12] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 13] 10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 14]10 is an explanatory diagram of an example of a procedure for determining the size of a marker using an instruction set included in a marker module 30G. FIG. [Figure 15] 10A and 10B are diagrams showing a modified example of a marker and another example of how the marker image points to the target object 5B. [Figure 16] FIG. 10 is an explanatory diagram showing a case where tilt correction is required. [Figure 17] FIG. 10 is a flow diagram of a marker display method. DETAILED DESCRIPTION OF THE INVENTION

[0011] The first controller of the present disclosure is a controller for a transmissive display device, and includes a memory, a processor, and a program stored in the memory and configured to be executable by the processor, the program including an instruction (first instruction) to acquire the relative positions of an observer's eyes, an object, and the transmissive display device positioned between the observer and the object, an instruction (second instruction) to calculate the binocular parallax of the observer when gazing at the object from the relative positions, and an instruction (third instruction) to determine the size of a marker to be displayed on the transmissive display device to indicate the object.

[0012] One of the features of the above controller is that it actively utilizes the fact that the marker is perceived as two (double) due to the observer's binocular parallax, and applies this to pointing to the target object. Conventionally, it has been thought that the double perception of images on a transmissive display device hinders accurate information transmission (pointing to an object using a marker). The inventors have conducted extensive research without being bound by such common technical knowledge, and have come up with the above-mentioned features.

[0013] Specifically, the controller includes in its program an instruction (third instruction) for determining the size of the marker (the entity displayed on the transmissive display device) based on the calculated binocular parallax. That is, the image of the marker that is perceived as double is adjusted and displayed so that it overlaps with the object. By doing so, when the observer gazes at the object, the image displayed on the transmissive display device is out of focus and appears double. By taking advantage of this, the observer can accurately point to the object using the image of the marker that appears as double.

[0014] A second controller of the present disclosure is a controller in which, in the first controller, the second instruction includes an instruction to calculate the binocular disparity based on information regarding the observer's right gaze and information regarding the observer's left gaze.

[0015] The observer's right and left gaze can be calculated and updated in real time by tracking the relative positions (coordinates) of the observer's eyes and the object. Calculating binocular disparity based on this and reflecting it in the size of the marker not only enables more accurate instructions, but also makes it easier to make the marker display follow the observer's posture, movement, etc.

[0016] A third controller of the present disclosure is a controller in which, in the first controller, the third instruction includes a fourth instruction to determine the size so that a first region is created in which at least a portion of the images of the marker that are perceived as two by the observer due to the binocular parallax overlap.

[0017] The area (first area) where the two visually recognized marker images overlap and the object are also visually recognized as overlapping, so that the indicated object becomes clearer to the observer.

[0018] A fourth controller of the present disclosure is a controller in which, in the third controller, the fourth instruction includes an instruction to determine that second areas defined by the image overlap to create the first area, or an instruction to determine that third areas indicated by the image overlap to create the first area.

[0019] For example, if the marker is a figure that includes a closed part, the area that overlaps with the area defined by this closed figure (second area) can be set as the first area, making it easier for the observer to identify the object being pointed to.

[0020] Furthermore, if the marker is a figure that includes a portion that opens toward the outer edge, the area where the surrounding area of ​​this figure (the indicated area, also called the "third area") overlaps can be set as the first area, making it easier for the observer to identify the indicated object. The shape and size of the indicated area depend on the shape of the marker, etc., but may be, for example, a circle surrounding the marker.

[0021] A fifth controller of the present disclosure is a controller according to the first to fourth controllers, wherein the program includes instructions for determining the display position of the marker on the display surface based on information regarding the observer's binocular line of sight and the position of the display surface of the transmissive display device.

[0022] As already explained, by determining the display position of the marker based on the line of sight of the observer with both eyes and the position of the display surface, the object being pointed to tends to become clearer to the observer.

[0023] A sixth controller of the present disclosure is the fifth controller, wherein the third instruction includes an instruction to determine the size of the marker so as to satisfy the formula: A / 2+B / 2≦X / 2≦A / 2+B, where B is the width of the object, A is the binocular disparity, and X is the width of the marker.

[0024] By setting X / 2 to A / 2+B (lower limit) or more, it becomes easy to generate the first region of the marker and display it so that it overlaps the object. Also, by setting X / 2 to A / 2+B (upper limit) or less, it is possible to achieve an excellent balance between the size of the object and the first region, making it easier for the observer to clearly see the object being pointed to.

[0025] A seventh controller of the present disclosure is a controller in which, in the first controller, the program includes instructions to correct the size and / or shape of the marker based on information regarding the observer's binocular line of sight and the attitude of the display surface of the transmissive display device.

[0026] The angle between the virtual line that defines the observer's line of sight and the display surface indicates the tilt of the transmissive display device relative to the observer. By correcting the size and / or shape of the marker based on this tilt, the image of the marker that the observer sees becomes closer to the actual shape. This makes it easier for the observer to identify the indicated object.

[0027] A first system of the present disclosure is a system including any one of first to fourth controllers and the transmissive display device.

[0028] The above system enables the marker display by the controller. By using this system, the observer can accurately point to the object using the image of the marker by gazing at the object and taking advantage of the fact that the image displayed on the transmissive display device is out of focus (double vision).

[0029] A first method of the present disclosure is a method for displaying a marker on a transmissive display device placed between an observer and an object to indicate the object, the method including: acquiring the relative positions of the observer's eyes, the transmissive display device, and the object; calculating the binocular disparity of the observer when gazing at the object from the relative positions; and determining the size of the marker to be displayed on the transmissive display device based on the binocular disparity.

[0030] According to the above method, the observer gazes at the object, and by taking advantage of the fact that the image displayed on the transmissive display device is out of focus (double vision), the observer can accurately point to the object using the image of the marker.

[0031] A first program of the present disclosure is a program that causes a computer to display a marker on a transmissive display device placed between an observer and an object to indicate the object, and includes an instruction (first instruction) to acquire the relative positions of the observer's eyes, the transmissive display device, and the object, an instruction (second instruction) to calculate the binocular disparity of the observer when gazing at the object from the relative positions, and an instruction (third instruction) to determine the size of the marker to be displayed on the transmissive display device based on the binocular disparity.

[0032] The program includes instructions for causing a computer to carry out the above-described method. By storing the medium storing the program in the memory of a computer and having a processor execute the program, the computer can be used as a controller for a transmissive display device. By loading the medium storing the program into a computer and using it as a controller for a transmissive display device, the object can be accurately indicated by the image of the marker, taking advantage of the fact that when an observer gazes at an object, the displayed image of the transmissive display device appears out of focus and double.

[0033] The controller and other components of the present disclosure will be described in detail below. The following description of the components may be based on representative embodiments, but the technical concept of the present disclosure is not limited to the embodiments. In this specification, a numerical range expressed using "to" means a range that includes the numerical values ​​before and after "to" as the lower and upper limits.

[0034] 1 is an explanatory diagram of a usage scene of a system including a transmissive display device controlled by a controller of the present disclosure. A transmissive display device 10 embedded in a wall surface 3 is placed in front of the eyes of an observer 1. As seen from the observer 1, multiple objects 5A and 5B are placed on shelves 7A and 7B behind the transmissive display device 10. The observer 1 can gaze at either the objects 5A or 5B through the transmissive display device 10.

[0035] In the following, explanations may be given based on the directions indicated on the axes in the drawings. The x-axis represents the depth direction (rear, front) as seen by the observer 1, the y-axis represents the lateral direction (right, left) as seen by the observer 1, and the z-axis represents the height direction (up, down) as seen by the observer 1. Similar axes may be indicated in other drawings as appropriate.

[0036] The transmissive display device 10 is a display device and is configured using a known optically transmissive liquid crystal panel, an optically transmissive (organic) electroluminescence (EL) panel, or the like. The transmissive display device 10 can display any video, image, character, or combination thereof (hereinafter also referred to as "image, etc.") at any location on its display surface 10A. The transmissive display device 10 is configured to transmit at least a portion of visible light, allowing the rear side to be seen through. In other words, the transmissive display device 10 is a display device that allows the observer 1 to visually recognize (through its display surface 10A) external light (from the back) and image light superimposed on each other. The transmissive display device 10 may be configured to output sound, etc., in addition to images, etc.

[0037] The transmissive display device 10 is a device that displays one image and / or video to be viewed by both eyes of an observer 1 on one display surface 10A.

[0038] The transmissive display device 10 is connected to a controller via a wired or wireless connection. The controller is a computer that controls the type, display timing, display position, size, etc. of images to be displayed on the display surface 10A of the transmissive display device 10.

[0039] Next, a system (display system) including the transmissive display device 10 will be described. Fig. 2 is a block diagram of a system including a first embodiment of the controller of the present disclosure. The system 100 includes the transmissive display device 10, a camera 12, and a controller 14.

[0040] The camera 12 is placed at a position where it can capture images of at least both eyes of the observer 1. The camera 12 may also be placed at a position where it can capture images of the transmissive display device 10 (the display surface 10A) and / or the objects 5A and 5B.

[0041] Furthermore, the system 100 includes one camera 12, but may include two or more cameras. In this case, at least one camera 12 is placed at a position where it can capture images of both eyes of the observer 1. The other cameras 12 may be placed at positions where it can capture images of both eyes of the observer 1, the transmissive display device 10, and / or the objects 5A and 5B.

[0042] Furthermore, the camera 12 of the system 100 is configured to capture images of the eyes of the observer 1 from a fixed state. However, the system 100 may also include a drive mechanism for the camera 12. The drive mechanism may be used to change the attitude of the camera 12, so that one or more cameras 12 can capture images of the eyes of the observer 1, the transmission-type display device 10, and / or the objects 5A and 5B.

[0043] The camera 12 may be a depth camera. A depth camera is an imaging device capable of acquiring depth-accompanying images including depth information. The depth camera is a TOF (Time Of Flight) imaging device that acquires depth information of an object by irradiating near-infrared light. The distance measurement method of the depth camera is not limited to the above, and a stereo camera method consisting of two imaging elements may also be used. Furthermore, a device that combines a distance measurement device that irradiates near-infrared light with an imaging device (other than a depth camera) can also be used as the camera 12.

[0044] The controller 14 is a control device for the transmissive display device 10. The controller 14 includes a processor 20, a memory controller 22, an I / O interface 24, an I / O controller 26, an I / O subsystem 28, and a memory 30. The transmissive display device 10 and the camera 12 are connected to the controller 14 by wire or wirelessly. Each part communicates with each other via a communication bus or a signal line.

[0045] Memory 30 may include random access memory (RAM), magnetic disk storage devices, and non-volatile memory devices such as flash memory devices.

[0046] Memory 30 may also include remotely located storage devices that may be accessed via RF circuitry, an external port, a communications network, or a combination thereof, not shown, that may be included in controller 14. Communications networks may include the Internet, an intranet, a local area network (LAN), a wide local area network (WLAN), a storage area network (SAN), and the like.

[0047] Access to memory 30 by components of controller 14, including processor 20, is controlled by memory controller 22. Note that processor 20, memory controller 22, etc. may be implemented on a single chip, or may be implemented separately on multiple chips.

[0048] The I / O subsystem 28 includes an I / O interface 24 and an I / O controller 26. The I / O subsystem 28 transmits and receives electrical signals for controlling the transmissive display device 10 and the camera 12.

[0049] The transmissive display device 10 displays a visual output to the observer 1. This visual output may include text, images (still images), video (moving images), and combinations thereof.

[0050] The controller 14 may include RF circuitry. The RF circuitry transmits and receives electromagnetic waves. The RF circuitry converts between electrical signals and electromagnetic waves, and may communicate with communication networks and other communication devices via these waves. Examples of RF circuitry include antenna systems, RF transceivers, amplifiers, tuners, oscillators, digital signal processors, CODEC chipsets, and memory, as well as other known circuitry for performing these functions.

[0051] The memory 30 stores software components including instructions or collections thereof (instruction sets, hereinafter also referred to as "modules") that can be executed by the processor 20. The software components include an operating system 30A, a camera control module 30B, a relative position detection module 30D, a binocular disparity calculation module 30E, and a graphics module 30F. In addition to the above, the software components may also include, for example, application modules.

[0052] Operating system 30A includes various software components and / or drivers for controlling general system tasks such as memory control and power management, and is responsible for communication between the hardware and software components included in controller 14.

[0053] The camera control module 30B may include a command set for detecting the eyes of the observer 1 by the camera 12, capturing images, etc. The camera control module 30B includes a pupil position detection module 30C as a sub-module. The pupil position detection module 30C may include a command set for detecting the eyes (pupils) of the observer 1 from the field of view of the camera 12. It should be noted that a publicly known algorithm for pupil position detection can be used.

[0054] The relative position detection module 30D includes an instruction set for detecting the relative positions (coordinates) of the eyes of the observer 1, the display surface 10A of the transmissive display device 10, and the objects 5A and 5B. Specifically, the module may include an instruction for calculating the position coordinates of the eyes of the observer 1 relative to the camera 12 based on the captured image and distance measurement data of the camera 12, and an instruction for converting the coordinates of the eyes of the observer 1 into the coordinate system of the display surface 10A of the transmissive display device 10 and the objects 5A and 5B stored in advance in the memory 30.

[0055] The relative position detection module 30D includes an instruction set for executing an instruction (first instruction) to acquire the relative positions of the eyes of the observer 1, the objects 5A and 5B, and the transmissive display device 10 arranged between the observer 1 and the objects 5A and 5B.

[0056] Furthermore, when the display surface of the transmissive display device 10 and the objects 5A and 5B are captured by the camera 12, which is a depth camera or a camera that also uses a distance measuring device, the relative position detection module 30D may include a command to detect the respective positions from these images. In other words, the positions of the display surface 10A of the transmissive display device 10 and / or the objects 5A and 5B do not need to be stored in advance in a memory.

[0057] When detecting the display surface of the transmissive display device 10 and the positions of the objects 5A and 5B from the image captured by the camera 12, a pattern (such as an "ArUco marker") may be attached to the transmissive display device 10 and / or the objects 5A and 5B in advance. The relative position detection module 30D may include a command to calculate the positions and orientations of the display surface of the transmissive display device 10 and the objects 5A and 5B from the position and orientation of the pattern. The detection of the relative positions may be performed and updated at regular intervals.

[0058] The binocular disparity calculation module 30E includes an instruction set for calculating the binocular disparity of the observer 1 based on the relative positions (coordinates) of the eyes of the observer 1 detected by the relative position detection module 30D, the display surface 10A of the transmissive display device 10, and the objects 5A and 5B. The binocular disparity calculation module 30E includes an instruction set for executing a second instruction for calculating the binocular disparity XA of the observer 1 when gazing at the objects 5A and 5B from the relative positions.

[0059] Fig. 3 is an explanatory diagram of the procedure for calculating the binocular disparity XA using a command set included in the binocular disparity calculation module 30E. Fig. 3 is a schematic diagram showing the observer 1, the transmissive display device 10, and the object 5B arranged in order from top to bottom on the paper (in the x-axis depth direction), viewed from above (from the front to the back of the paper, downward in the z-axis direction).

[0060] As described above, the left and right pupils 2R and 2L are detected by the pupil position detection module 30C from the image of the eyes 1R and 1L (both eyes) of the observer 1 captured by the camera 12. Next, the positions (coordinates) of the pupils 2R and 2L of the observer 1 are detected by the instruction set included in the relative position detection module 30D. Furthermore, the positions (coordinates) of the transmissive display device 10 and the object 5B and the positions (coordinates) of the pupils 2R and 2L are calculated in a unified coordinate system by coordinate transformation (the relative positions are calculated). Note that in this example, the pupils 2R and 2L of the observer 1 are the positions of the observer's both eyes, but this is not limited to the above. The positions of the eyes 1R and 1L can be determined by any method depending on the image resolution, etc. In the following description, the pupils 2R and 2L of the observer 1 are the positions of the observer's both eyes.

[0061] The binocular disparity calculation module 30E may include the following instructions: an instruction to calculate, from the relative positions, a midpoint MP1 of the line segment connecting the pupils 2R and 2L, an instruction to calculate a virtual line (first straight line SL1) passing through the midpoint MP1 and the object 5B, an instruction to calculate an intersection (first intersection IP1) between the first straight line SL1 and the display surface 10A of the transmissive display device 10, an instruction to calculate two lines (second straight line SL2) passing through the pupils 2R, 2L and the first intersection IP1, an instruction to calculate an intersection (second intersection IP2) between the second straight line SL2 and the reference plane 11 including the object 5B, and an instruction to calculate the distance (binocular disparity XA) between the second intersection IP2.

[0062] Of these, the straight lines (second straight line SL2) passing through the pupils 2R and 2L and the first intersection point IP1 are information relating to the right and left gazes, respectively. That is, the second command of the binocular disparity calculation module 30E includes a command to calculate binocular disparity based on information relating to the observer's right gaze and information relating to the observer's left gaze.

[0063] In this case, the reference plane 11 is defined as, for example, a plane that is perpendicular to a virtual line (first straight line SL1) that connects the object 5B and the midpoint MP1 of the line connecting the pupils 2R and 2L and that includes the object 5B. The virtual line (first straight line SL1) corresponds to the line of sight of the observer's both eyes. Note that the reference plane 11 does not need to be perpendicular to the virtual line (first straight line SL1) as long as it intersects with it. Furthermore, the reference plane 11 can be defined so as not to intersect with the display surface 10A. For example, the reference plane 11 and the display surface 10A can be approximately parallel. Furthermore, the reference plane 11 and the line segment connecting the pupils 2R and 2L can be approximately parallel. The reference plane 11 can be selected as a plane that is approximately parallel to the display surface 10A and / or the line segment connecting the pupils 2R and 2L.

[0064] The graphics module 30F includes various software components for displaying graphics such as text, images, and videos on the transmissive display device 10. The graphics module 30F includes a marker module 30G and a tilt correction module 30H.

[0065] The marker module 30G includes an instruction set for determining, based on the calculated binocular disparity XA (distance A), the position, shape, and size of a marker to be displayed on the transmissive display device 10. The marker module 30G includes an instruction set for executing a third instruction for determining the size of a marker to be displayed on the transmissive display device 10 in order to indicate the object 5B, and an instruction set for executing an instruction for determining the display position of the marker on the display surface 10A based on information about the binocular line of sight of the viewer 1 and the position of the display surface 10A of the transmissive display device 10.

[0066] 4 to 14 are explanatory diagrams of an example of a procedure for determining the size of a marker using a command set included in the marker module 30G. Each diagram schematically shows an image visually recognized by the observer 1. Axes representing directions are attached to FIG. 4(a). Although not attached to the other diagrams, the axes representing directions in all of FIGS. 4 to 14 are the same as those in FIG. 4(a). For simplicity, the target object 5B is represented as a circle of width B.

[0067] 4(a) is a diagram showing (the actual object of) a marker 40 displayed on the display surface 10A of the transmissive display device 10. In FIGS. 4(b), 5(a), and 5(b), the marker image 40R and the marker image 40L viewed by the right eye of the observer 1 are circles of the same shape as in FIG. 4(a). The two marker images 40R and 40L are images of the actual marker 40 displayed on the transmissive display device 10, and in principle, have the same shape and size, and are shifted parallel (horizontally) from each other with respect to the line segment connecting the eyes (pupils 2R and 2L) of the observer 1.

[0068] When the marker 40 is displayed at the intersection (first intersection IP1) between the first straight line SL1 and the transmission type display device 10, the centers of the marker images 40R and 40L become the intersection (second intersection IP2R and IP2L) between the second straight lines SL2R and SL2L and the reference plane 11. The second straight lines SL2R and SL2L correspond to the right and left lines of sight of the observer 1, respectively. Here, in FIG. 4(b), the relationship between the distance A of the binocular parallax XA and the width B (diameter) of the object 5B is B. <Aとなっている。

[0069] The instruction set included in the marker module 30G includes an instruction to determine the display position of the marker 40 on the transmissive display device 10, and an instruction to determine the size of the marker so that the marker is displayed overlapping the object 5B when the observer 1 gazes at the object 5B.

[0070] The display position of the marker 40 on the transmissive display device 10 may be the intersection (first intersection IP1) of a first line SL1 (the line of sight of both eyes of the observer 1) calculated by a command included in the relative position detection module 30D and the position of the display surface of the transmissive display device 10, which intersection (first intersection IP1) is determined from the first line SL1 and the line of sight of the observer 1 gazing at the object 5B. By displaying the marker 40 at the intersection (first intersection IP1) with the first line SL1, the marker image can be visually recognized so as to overlap with the object 5B.

[0071] The procedure for determining the size of marker 40 is not particularly limited. For example, marker module 30G may include a command (third command) to determine the size of the marker, taking into account binocular parallax XA (i.e., distance A between the center points of marker images 40R and 40L), so as to generate an overlapping region (first region RZ1) between regions (second region RZ2) defined by marker image 40R and marker image 40L. Also, marker module 30G may include a command (fourth command) to determine the size of the marker, taking into account binocular parallax XA, so as to generate an overlapping region (first region RZ1) between regions (third region RZ3) defined by marker image 40R and marker image 40L.

[0072] Next, with reference to Figure 4(b) and Figures 5(a) and (b), we will explain an example of a procedure for determining the size of a marker so that an area (first area RZ1) is created where the areas (second area RZ2) defined by marker image 40R and marker image 40L overlap.

[0073] 4(b) is a schematic diagram of an image when the width X of the marker images 40R, 40L (i.e., the marker 40) is equal to the binocular parallax XA (distance A), i.e., X / 2 is equal to A / 2. In this case, the two marker images 40R, 40L are in contact at the center of the object 5B (third intersection point IP3). In this case, although there is no overlap between the areas defined by the marker images 40R and 40L, the observer 1 can visually recognize that the two marker images 40R, 40L and the object 5B overlap, and can recognize that the object 5B is being pointed to.

[0074] FIG. 5(a) is a schematic diagram of an image when the width X of the marker images 40R and 40L (i.e., the marker 40) is equal to the sum of the binocular parallax XA and the width B of the object 5B, i.e., X / 2 is equal to A / 2 + B / 2. In this case, an overlapping region (first region RZ1) is generated between the region (second region RZ2) defined by the marker images 40R and 40L. For ease of explanation, the first region RZ1 is shaded in the drawing. Furthermore, since the width of the first region RZ1 is equal to the width B of the object 5B, in the case of FIG. 5(a), the entire object 5B is included in the first region RZ1. In this way, the observer 1 visually recognizes the two marker images 40R and 40L and the object 5B overlapping with each other, and can more accurately recognize that the object 5B is indicated as being included in the first region RZ1.

[0075] FIG. 5(b) is a schematic diagram of an image when the width X of the marker images 40R and 40L (i.e., the marker 40) is equal to the sum of the binocular parallax XA and twice the width B of the object 5B, i.e., X / 2 is equal to A / 2+B. In this case, as in FIG. 5(a), an overlapping region (first region RZ1) is generated between the regions defined by the marker images 40R and 40L. In this case, the width of the first region RZ1 is larger than the width B of the object 5B. The entire object 5B is included in the first region RZ1, as in FIG. 5(a). On the other hand, because the marker images 40R and 40L are larger, when multiple objects 5B are lined up, the marker images 40R and 40L may interfere with one another.

[0076] From the above, regarding the width X of the marker images 40R and 40L (i.e., marker 40), as one form, the formula: A / 2 + B / 2 ≤ X / 2 ≤ A / 2 + B is preferably set to be satisfied. The instruction set included in the marker module 30G may include an instruction for determining X so as to satisfy the above relationship.

[0077] Note that the above relationship is the same whether the binocular parallax XA and the width B of the object 5B are equal (A = B) or the width B of the object 5B is larger than the binocular parallax XA (A < B). FIGs. 6(a), (b), and FIG. 7 are schematic diagrams of images in the case of A = B. FIG. 6(a) corresponds to FIG. 4(b) and is a schematic diagram of an image when the width X of the marker images 40R and 40L (i.e., marker 40) is made equal to the binocular parallax XA, that is, when X / 2 is made equal to A / 2.

[0078] Also, FIG. 6(b) corresponds to FIG. 5(a) and is a schematic diagram of an image when the width X of the marker images 40R and 40L (i.e., marker 40) is made equal to the sum of the binocular parallax XA (distance A) and the width B of the object 5B, that is, when X / 2 is made equal to A / 2 + B / 2.

[0079] Also, FIG. 7 corresponds to FIG. 5(b) and is a schematic diagram of an image when the width X of the marker images 40R and 40L (i.e., marker 40) is made equal to the sum of the binocular parallax XA (distance A) and twice the width B of the object 5B, that is, when X / 2 is made equal to A / 2 + B. [[ID=2⃣0]]

[0080] Also, FIGs. 8(a), (b), and FIG. 9 are schematic diagrams of images in the case of A < B.

[0081] FIG. 8(a) corresponds to FIG. 4(b) and is a schematic diagram of an image when the width X of the marker images 40R and 40L (i.e., marker 40) is made equal to the binocular parallax XA (distance A), that is, when X / 2 is made equal to A / 2.

[0082] Also, Figure 8(b) corresponds to Figure 5(a) and is a schematic diagram of the image when the width X of the marker images 40R, 40L (i.e., marker 40) is equal to the sum of the binocular parallax XA (distance A) and the width B of the object 5B, i.e., X / 2 is equal to A / 2 + B / 2.

[0083] Also, Figure 9 corresponds to Figure 5(b) and is a schematic diagram of the image when the width X of the marker images 40R, 40L (i.e., marker 40) is equal to the sum of the binocular parallax XA (distance A) and twice the width B of the object 5B, i.e., X / 2 is equal to A / 2+B.

[0084] It can be seen that in the cases of Figures 6(a), (b), and 7, and Figures 8(a), (b), and 9, the same visual effect as in Figure 4(b), and Figures 5(a), (b) can be obtained.

[0085] 4(b) and 5(a) and (b), the region (first region RZ1) formed by the overlap of marker images 40R and 40L is formed by the overlap of regions (second region RZ2) defined by marker images 40R and 40L, which are closed figures. On the other hand, the first region RZ1 may be formed by the overlap of regions (third region RZ3) indicated by the marker images.

[0086] 10 and 11 are diagrams showing the procedure for determining the size of the marker 44 when the marker 44 is a figure that opens toward the outer edge.

[0087] First, Fig. 10(a) is a diagram showing a physical marker 44 displayed on the display surface 10A of the transmissive display device 10. As shown in Fig. 10(a), the marker 44 is shaped like the letter "X" and opens from the center to the outer edge. In this case, it is not possible to conceive of an area defined by the visually recognized marker image, but it is possible to conceive of the existence of an area (third area RZ3) indicated by the marker image.

[0088] Although the third region RZ3 is originally an area indicated by a marker image, for convenience of explanation, its range is shown in FIG. 10(a) which shows the actual marker 44. The size and shape of the third region RZ3 may vary depending on the shape of the marker 44. As shown in the example of FIG. 10(a), the third region RZ3 may be defined as a closed figure surrounding the periphery of a figure. By overlapping the third region RZ3 to create the first region RZ1, the target object 5B can be more accurately indicated.

[0089] 10(b) and 11(a) and (b) are explanatory diagrams of an example of a procedure for determining the size of the marker so that the areas (third area RZ3) indicated by the marker image 44R and the marker image 44L overlap each other. <Aのケースである。

[0090] 10(b) is a schematic diagram of an image when the width X of the marker images 44R, 44L (i.e., the marker 44) is equal to the binocular parallax XA (distance A), i.e., X / 2 is equal to A / 2. In this case, the pointing regions (third region RZ3, not shown) of the two marker images 44R, 44L are in contact at the center (third intersection IP3) of the object 5B. In this case, although the pointing regions (third region RZ3) of the marker images 44R and 44L do not overlap, the observer 1 visually recognizes that the two marker images 44R, 44L and the object 5B overlap, and can recognize that the object 5B is being pointed to.

[0091] 11(a) is a schematic diagram of an image when the width X of the marker images 44R and 44L (i.e., the marker 44) is equal to the sum of the binocular parallax XA and the width B of the object 5B, i.e., X / 2 is equal to A / 2 + B / 2. In this case, an overlapping region (first region RZ1) is generated between the regions indicated by the marker images 44R and 44L (third region RZ3, not shown). Furthermore, since the width of the first region RZ1 is equal to the width B of the object 5B, in the case of FIG. 11(a), the entire object 5B is included in the first region RZ1.

[0092] In this way, for observer 1, the two marker images 44R and 44L overlap with the object 5B and are visually recognized, and furthermore, it can be more accurately recognized that the object 5B is indicated so as to be included in the first region RZ1.

[0093] FIG. 11(b) is a schematic diagram of an image when the width X of the marker images 44R and 44L (that is, the marker 44) is equal to the sum of the binocular parallax XA (distance A) and twice the width B of the object 5B, that is, when X / 2 is equal to A / 2 + B. In this case, similar to FIG. 11(a), a region (the first region RZ1) that overlaps with the regions (the third region RZ3, not shown) indicated by the marker image 44R and the marker image 44L respectively occurs. In this case, the width of the first region RZ1 becomes larger than the width B of the object 5B. The entire object 5B is in a form included in the first region RZ1 as in FIG. 11(a). On the other hand, since the marker images 44R and 44L become larger, when there are a plurality of objects 5B arranged, there may be a case where the other objects interfere with the marker images 44R and 44L.

[0094] From the above, regarding the width X of the marker images 44R and 44L (that is, the marker 44), as one form, Equation: A / 2 + B / 2 ≤ X / 2 ≤ A / 2 + B It is preferably set to satisfy. The instruction set included in the marker module 30G may include an instruction for determining X so as to satisfy the above relationship. Although only the case where B < A has been described, the above relationship is the same for the cases of A = B and A < B.

[0095] The above is a procedure for determining the sizes (widths) of the markers 40 and 44 such that the region (the second region RZ2) partitioned by the closed-shaped marker 40 or the region (the third region RZ3) indicated by the marker 44 having an open shape from the center toward the outer edge overlap to generate the first region RZ1.

[0096] However, the procedure for determining the width of the marker may be other than the above. For example, even if the first region RZ1 is not generated, adjustment (determination of the size of the marker) for instructing the object 5B can be performed. Hereinafter, an example of the procedure will be described.

[0097] FIGS. 12(a) and (b) are explanatory diagrams of a form in which an object is sandwiched by marker images 46R and 46L obtained by a rectangular marker.

[0098] FIG. 12(a) is a schematic diagram of an image when the width X of the marker images 46R and 46L is equal to the difference between the binocular parallax XA (distance A) and the width B of the object 5B, that is, when X / 2 is equal to A / 2 - B / 2. In this case, the respective ends of the marker image 46R and the marker image 46L are in contact with the object 5B, and are in a form that sandwiches the object 5B. By doing so, for the observer 1, it can be more accurately recognized that the object 5B is being indicated.

[0099] FIG. 12(b) is a schematic diagram of an image when the width X of the marker images 46R and 46L is equal to the difference between half of the binocular parallax XA (distance A) and half of the width B of the object 5B, that is, when X / 2 is equal to 1 / 2(A / 2 - B / 2). In this case, the respective ends of the marker image 46R and the marker image 46L are not in contact with the object 5B, and are in a form that sandwiches the object 5B. By doing so, for the observer 1, it can be more accurately recognized that the object 5B is being indicated. In this case, the closer the distance between the marker images 46R and 46L and the object 5B, the more accurately the observer 1 can recognize that the object 5B is being indicated.

[0100] From the above, regarding the width X of the marker images 46R and 46L, as one form, Formula: 1 / 2(A / 2 - B / 2) ≤ X / 2 ≤ A / 2 - B / 2 It is preferably set to satisfy. The instruction set included in the marker module 30G may include an instruction for determining X so as to satisfy the above relationship. Note that the above is the case where B < A. Hereinafter, the cases of A = B and A < B will also be described.

[0101] Figures 13(a) and (b) are explanatory diagrams of a form in which an object is sandwiched by marker images 46R and 46L obtained by a rectangular marker when A = B. In this case, as shown in the figure, regarding the width X of the marker images 46R and 46L, as one form, it is preferably set to satisfy the formula: B / 4 ≦ X ≦ B / 2.

[0102] When the width X of the marker images 46R and 46L is greater than or equal to the lower limit value, the visibility of the marker images 46R and 46L is high, and the observer 1 can recognize the target object 5B more accurately. On the other hand, when the width X of the marker images 46R and 46L is less than or equal to the upper limit value, it is easier for the observer 1 to recognize that the object 5B is "sandwiched" by the marker images 46R and 46L. As a result, the observer 1 can recognize the target object 5B more accurately.

[0103] Also, Figures 14(a) and (b) are explanatory diagrams of a form in which an object is sandwiched by marker images 46R and 46L obtained by a rectangular marker when A < B. In this case, as shown in the figure, regarding the width X of the marker images 46R and 46L, as one form, it is preferably set to satisfy the formula: B - A ≦ X ≦ B / 2.

[0104] When the width X of the marker images 46R and 46L is greater than or equal to the lower limit value, the visibility of the marker images 46R and 46L is high, and the observer 1 can recognize the target object 5B more accurately. On the other hand, when the width X of the marker images 46R and 46L is less than or equal to the upper limit value, it is easier for the observer 1 to recognize that the object 5B is "sandwiched" by the marker images 46R and 46L. As a result, the observer 1 can recognize the target object 5B more accurately.

[0105] The above is an example of a set of instructions for determining the size of the marker based on the binocular parallax XA included in the marker module 30G. The determination of the size of the marker can adopt various variations depending on the shape of the marker and the indication form.

[0106] FIG. 15 is a diagram showing a modified example of the marker and another example of how the marker image points to the target object 5B.

[0107] FIG. 15(a) shows a rectangular marker 50 (left diagram). The size of the marker 50 may be determined so that the area (first area RZ1) defined by the marker images 50R and 50L surrounds the target object 5B (middle diagram), or the size of the marker 50 may be determined so that the marker images 50R and 50L sandwich the target object 5B (right diagram). Note that in FIG. 15(a) , the marker images 50R and 50L are all depicted as having the same size in the diagrams showing the respective pointing forms, but this is for convenience of explanation. In reality, the size (width X) of the marker is appropriately determined based on the binocular parallax XA. The binocular parallax XA is actually different between the middle diagram and the right diagram of FIG. 15(a), and this is not intended to explain how the size is determined, but simply to explain variations in the pointing form.

[0108] Similarly to the above, when using markers 51, 52, and 53 shown in Figures 15(b), 15(c), and 15(d), respectively, the marker images 51R, 51L, 52R, 52L, 53R, and 53L can be used to superimpose the second region RZ2 (not shown) or the third region RZ3 (not shown) to create the first region RZ1 (middle figure), or a pointing form can be adopted in which the target object 5B is sandwiched between the marker images 51R, 51L, 52R, 52L, 53R, and 53L.

[0109] Returning to FIG. 2, the tilt correction module 30H included in the graphics module 30F stored in the memory of the system 100 includes an instruction set for correcting the size and / or shape of the marker based on the angle between the first straight line SL1 and the transmissive display device 10.

[0110] Fig. 16 is an explanatory diagram of a case where tilt correction is required. In Fig. 16(a), the viewer 1 is facing the display surface of the transmissive display device 10 almost directly. In this case, the marker 42A (entity) displayed on the display surface of the transmissive display device 10 is viewed by the viewer 1 in its original shape (although it may appear double). On the other hand, in the case of Fig. 16(b), the display surface of the transmissive display device 10 is tilted toward the rear (x-axis direction) with respect to the line of sight (first straight line SL1) of the viewer 1, and the marker 42 (entity) viewed by the viewer 1 is distorted vertically (in the tilted direction).

[0111] The tilt correction module 30H includes instructions to correct the shape based on information about the line of sight of both eyes of the viewer 1 and the attitude of the display surface 10A. The instruction set includes instructions to perform calculations according to the following procedure and calculate the amount of correction.

[0112] First, if the vector of the first straight line SL1 (corresponding to the line of sight of both eyes) is (Ax, Ay, Az) and the normal vector of the display surface 10A of the transmissive display device 10 (corresponding to the attitude of the display surface 10A) is (Bx, By, Bz), the angle θx of the line of sight in the X-axis direction and the angle θy of the line of sight in the Y-axis direction as seen by the observer 1 can be calculated using the following equations.

number

[0113] From θx and θy, the shrinkage (Dx, Dy) of the marker due to the tilt of the transmission type display device 10 can be calculated by the following formula.

number

[0114] The missing Dx and Dy are added to the range in the X-axis direction and the Y-axis direction, respectively, but the influence of the tilt of the display surface 10A at this time is corrected, and the range correction values ​​(Δx, Δy) are calculated.

number

number

[0115] In practice, the effect of deformation of the marker image due to tilt of the display surface 10A of the transmissive display device 10 may not be significant. Therefore, tilt correction is not essential, and the graphics module 30F stored in the memory of the system 100 does not need to include the tilt correction module 30H.

[0116] For example, when a marker with a diameter of 10 cm is used, if the transmission type display device 10 is tilted by 10 degrees in the X-axis direction as seen by the observer 1, the minor axis of the marker image seen by the observer 1 will appear to be 9.858 cm. As a result, the minor axis is 0.142 cm (a reduction of 1.142%).

[0117] Next, a description will be given of a method (marker display method) that can be implemented by the above-described system 100. Fig. 17 is a flow chart of the marker display method.

[0118] In step S10, the method captures an image of the face of the observer 1 with the camera 12 and detects the three-dimensional positions of both eyes (pupils 2L, 2R) of the observer 1. In this step, the processor 20 executes instruction sets included in the camera control module 30B and pupil position detection module 30C to control the camera 12.

[0119] Next, in step S11, the midpoint MP1 of the line segment connecting both eyes (pupils 2R, 2L) of the observer 1 is calculated, and a straight line (first straight line SL1) passing through the object 5B and the midpoint MP1 is calculated. In this step, the processor 20 executes a set of instructions included in the relative position detection module 30D.

[0120] Next, in step S12, the intersection (first intersection IP1) between the first straight line SL1 and the display surface 10A of the transmissive display device 10 is calculated. Next, in step S13, a line (second straight line SL2) passing through each of the eyes (pupils 2R, 2L) of the viewer 1 and the first intersection IP1 is calculated. Next, in step S14, the intersection (second intersection IP2) between the second straight line SL2 and the reference plane 11 including the object 5B is calculated. Next, in step S15, the distance A (binocular parallax XA) between the second intersection IP2 is calculated. In these steps, the processor 20 executes a set of instructions included in the binocular disparity calculation module 30E.

[0121] Next, in step S16, the shape and size of the marker are determined based on the distance A (binocular parallax XA). In this step, the processor 20 executes the instruction sets included in the graphics module 30F and the marker module 30G.

[0122] Next, in step S17, the shape and size of the marker are corrected based on the angle between the display surface 10A of the transmissive display device 10 and the first line SL1. In this step, the processor 20 executes the instruction sets included in the graphics module 30F and the tilt correction module 30H.

[0123] Next, in step S18, a marker is displayed on the display surface 10A of the transmissive display device 10. In this step, the processor 20 executes the instruction set included in the graphics module 30F.

[0124] Note that steps S12 to S15 in the above method are one example of a method for calculating the binocular disparity XA, and a method other than the above may be used for calculating the binocular disparity XA. Also, step S17 in the above method is not essential and may not be included. [Explanation of symbols]

[0125] 1. Observer 1R, 1L eye 2R, 2L Pupils 3 Wall 5A, 5B Objects 7A, 7B shelves 10 Transparent display device 10A display surface 11 Reference plane 12 Camera 14 Controller 20 processors 22 Memory Controller 24 I / O interfaces 26 I / O controller 28 I / O Subsystem 30 memory 30A Operating System 30B Camera Control Module 30C Pupil Position Detection Module 30D Relative Position Detection Module 30E Binocular Disparity Calculation Module 30F Graphics Module 30G Marker Module 30H Correction Module 40, 44, 42A, 50, 51, 52, 53 markers 40L, 40R, 44L, 44R, 46L, 46R, 50L, 50R, 51L, 51R, 52L, 52R, 53L, 53R Marker images 100 systems XA binocular parallax IP1 1st intersection IP2, IP2L, IP2R 2nd intersection IP3 3rd intersection MP1 midpoint RZ1 1st area RZ2 2nd area RZ3 3rd area SL1 First Straight Line SL2, SL2L, SL2R Second Straight Line

Claims

1. A controller for a transmissive display device, A memory, a processor, and a program stored in the memory and configured to be executable by the processor, The program includes a first command for acquiring relative positions of both eyes of an observer, an object, and the transmission-type display device disposed between the observer and the object; a second command to calculate a binocular disparity of the observer when gazing at the object from the relative positions; and third instructions for determining a size of a marker to be displayed on the transmissive display device to indicate the object.

2. The controller according to claim 1 , wherein the second instruction includes an instruction to calculate the binocular disparity based on information about the observer's right line of sight and information about the observer's left line of sight.

3. The controller of claim 1 , wherein the third instruction includes a fourth instruction for determining the size so as to create a first region in which at least a portion of the images of the marker that are perceived as two by the observer due to the binocular parallax overlap.

4. 4. The controller of claim 3, wherein the fourth instruction includes an instruction to determine that second areas defined by the images overlap to form the first area, or an instruction to determine that third areas indicated by the images overlap to form the first area.

5. The controller according to claim 1 , wherein the program includes an instruction to determine the display position of the marker on the display surface based on information about the observer's binocular line of sight and the position of the display surface of the transmissive display device.

6. The third command is, when the width of the object is B, the binocular parallax is A, and the width of the marker is X, Formula: A / 2+B / 2≦X / 2≦A / 2+B The controller of claim 5 , further comprising instructions for determining the size of the marker to satisfy:

7. The controller according to claim 1 , wherein the program includes an instruction to correct the size and / or shape of the marker based on information about the viewer's binocular line of sight and the attitude of the display surface of the transmissive display device.

8. A system comprising the controller according to any one of claims 1 to 4 and the transmissive display device.

9. A method for displaying a marker on a transmissive display device disposed between a viewer and an object to indicate the object, comprising: acquiring relative positions of the observer's eyes, the transmission type display device, and the object; calculating a binocular disparity of the observer when gazing at the object from the relative positions; determining a size of the marker to be displayed on the transmissive display device based on the binocular disparity.

10. On the computer, A program for displaying a marker on a transmission type display device disposed between an observer and an object to indicate the object, a first command for acquiring relative positions of the observer's eyes, the transmission type display device, and the object; a second command to calculate a binocular disparity of the observer when gazing at the object from the relative positions; and a third instruction for determining a size of the marker to be displayed on the transmissive display device based on the binocular parallax.

Images