Eyeglass-type display device and display system

Abstract

This eyeglass-type display device comprises a transparent display unit, a specification unit, and a display control unit. The specification unit specifies a focus area, which includes a real object gazed at by a user within the user's field of view through the display unit, on the basis of voices spoken by the user of the eyeglass-type display device. The display control unit displays a virtual object corresponding to the real object gazed at by the user on a display area that overlaps with the focus area within the user's field of view through the display unit.

Description

Technical Field

[0004] ,

[0001] The present invention relates to a glasses-type display device and a display system.

Background Art

[0002] An HMD (Head Mounted Display) device that displays a virtual object representing additional information such as a description text related to a real object overlaid on the real object (hereinafter, real object) in the real-world view is generally known. Among HMD devices, there are also those that display virtual objects overlaid on the real space without blocking the user's field of view, such as AR (Augmented Reality) glasses or MR (Mixed Reality) glasses (for example, Patent Document 1 and Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] HMD devices that display virtual objects without obstructing the user's field of view have a problem: if there is unwanted light in the area containing the real object the user is focusing on (hereinafter referred to as the "area of ​​focus"), the sense of immersion or convenience may be impaired. To solve this problem, one could consider using an attachment that obstructs the entire field of view seen by the user through the HMD device. However, if the entire field of view is always obstructed, the user will be unable to visually grasp the situation in the surrounding real space. Another option is to control the display by blocking light from the real space only in the area of ​​focus specified by the user using a mouse or keyboard, and then displaying virtual objects. However, using a mouse or keyboard to specify the area of ​​focus would reduce user convenience. [Means for solving the problem]

[0005] A spectacle-type display device having a transparent display unit for displaying virtual objects, according to a preferred embodiment of the present disclosure, comprises a identification unit and a display control unit. The identification unit identifies an area of ​​interest in the user's field of view that includes a real object that the user is fixated on, based on the voice spoken by the user. The display control unit causes the virtual object corresponding to the real object to be displayed in the display unit in the display area that overlaps with the area of ​​interest in the field of view.

[0006] A display system according to a preferred embodiment of the present disclosure comprises a spectacle-type display device worn on the user's head and having a transparent display unit on which virtual objects are displayed; a identification unit; and a display control unit. The identification unit identifies an area of ​​interest in the user's field of view that includes a real object that the user is fixated on, based on the voice spoken by the user. The display control unit causes the virtual object corresponding to the real object to be displayed in the display area of ​​the display unit that overlaps with the area of ​​interest in the field of view. [Effects of the Invention]

[0007] According to this disclosure, since the area of ​​interest is identified based on the user's voice, a virtual object can be displayed overlaid on the area of ​​interest without compromising user convenience. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of a display system 1 according to one embodiment of the present disclosure. [Figure 2] This figure shows an example of poster P where actual objects are placed. [Figure 3] This figure shows an example of a management table TBL in this disclosure. [Figure 4] This is a block diagram showing an example configuration of a mobile device 10. [Figure 5] This flowchart shows the flow of the display method executed by the processing unit 18 of the mobile device 10 according to program PR1. [Figure 6] This is a block diagram showing an example configuration of the eyeglass-type display device 20. [Figure 7] This is a diagram illustrating the operation of this embodiment. [Figure 8] This is a diagram illustrating the operation of this embodiment. [Figure 9] This figure shows an example of the image that user U sees in variation 1. [Figure 10] This figure shows an example of the image that user U sees in modified example 2. [Modes for carrying out the invention]

[0009] A. Embodiment Figure 1 is a block diagram showing an example configuration of a display system 1 according to one embodiment of the present disclosure. As shown in Figure 1, the display system 1 includes a portable device 10 and a glasses-type display device 20. The glasses-type display device 20 is worn on the head of user U. The glasses-type display device 20 is an HMD device that displays virtual objects that do not exist in real space without obstructing the field of view of user U wearing the glasses-type display device 20. The glasses-type display device 20 has an imaging function. The glasses-type display device 20 worn on the head of user U captures an image of real space corresponding to user U's field of view using the imaging function. The image captured by the glasses-type display device 20 worn on the head of user U shows objects that exist in user U's field of view, i.e., real objects.

[0010] The portable device 10 is, for example, a smartphone. The portable device 10 is worn on the user U's body. The portable device 10 is worn on the user U's body by hanging it around the neck using a strap or the like. The portable device 10 has a sound pickup function. The portable device 10 worn on the user's body picks up the voice emitted by the user U using the sound pickup function. The portable device 10 is also connected by wire to a glasses-type display device 20 worn on the head. The portable device 10 may also be connected to the glasses-type display device 20 wirelessly. The portable device 10 acquires image data representing the image captured by the glasses-type display device 20 from the glasses-type display device 20. Note that the portable device 10 is not limited to a smartphone; for example, it may be a tablet or a notebook personal computer.

[0011] Furthermore, the mobile device 10 communicates with the management device 30 via a communication network NW. The mobile device 10 transmits image data acquired from the glasses-type display device 20 to the management device 30. The management device 30 is a server device that provides location recognition services and content management services in AR.

[0012] The location recognition service is a service that identifies the location of the glasses-type display device 20 in a global coordinate system based on images captured by the imaging function of the glasses-type display device 20. Specific implementation methods of the location recognition service include using AR tags or utilizing the distribution of feature points extracted from images, such as SLAM (Simultaneous Localization and Mapping). The content management service is a service that delivers information to the glasses-type display device 20 regarding virtual objects corresponding to one or more real objects visible from the location of the glasses-type display device 20 in a global coordinate system.

[0013] The management device 30 stores, in advance, virtual object information representing an image of a virtual object, and area information indicating the position and size of the display area for displaying the virtual object, for each virtual object corresponding to one or more real objects visible from a position in the global coordinate system. The management device 30 identifies one or more real objects from the position of the glasses-type display device 20, which it has identified based on image data received from the mobile device 10 via the communication network NW. It then sends back the virtual object information and area information corresponding to each of the identified one or more real objects to the mobile device 10. The mobile device 10 displays the image of the virtual object on the glasses-type display device 20 according to the virtual object information and area information received from the management device 30. As a result, the virtual object is superimposed on the real space in the eyes of the user U.

[0014] The real space in this embodiment is, for example, the venue of a poster session at a research presentation such as a conference. The real object in this embodiment is, for example, each English word in a poster posted in the venue of a poster session and having its research content described in English. FIG. 2 is a diagram showing an example of a poster P posted in the venue of a poster session. The virtual object in this embodiment is a character string representing the Japanese translation of the English words described in the poster P. In this embodiment, the management device 30 transmits virtual object information and area information to the portable device 10 by transmitting a management table TBL shown in FIG. 3 to the portable device 10.

[0015] As shown in FIG. 3, in the management table TBL, virtual object information and area information about the virtual object corresponding to the real object are stored in association with the identification information for identifying the real object. The identification information in this embodiment is data of a character string representing the pronunciation of the real object identified by the identification information. For example, assuming that the English word "Patients" is the real object. The identification information in this case is the katakana "ペイシェンツ". Katakana are phonetic characters used in Japanese to represent the pronunciation of foreign words. The virtual object information corresponding to this identification information represents an image of the Japanese character string "病人" representing the Japanese translation of the English word "Patients". In the image represented by the virtual object information, the background of the Japanese character string is filled with a predetermined color such as white to block the light from the real space. Also, the area information stored in the management table TBL in association with the identification information "ペイシェンツ" represents the area overlapping with the real object "Patients".

[0016] FIG. 4 is a block diagram showing a configuration example of the mobile device 10. As shown in FIG. 4, the mobile device 10 includes an input device 11, an output device 12, a microphone 13, a communication device 14, a communication device 15, a storage device 17, a processing device 18, and a bus 19. The input device 11, the output device 12, the microphone 13, the communication device 14, the communication device 15, the storage device 17, and the processing device 18 are interconnected by a bus 19 for communicating information. The bus 19 may be configured using a single bus or may be configured using different buses for each device.

[0017] The input device 11 includes a touch panel. The input device 11 may include a plurality of operation keys in addition to the touch panel. The input device 11 may include a plurality of operation keys without including the touch panel. The input device 11 receives an operation performed by the user U. The output device 12 includes a display. The touch panel of the input device 11 is laminated on the display of the output device 12. The output device 12 displays various information.

[0018] The microphone 13 picks up the voice of the user U. The microphone 13 generates sound data indicating the waveform of the picked-up sound and outputs it to the processing device 18. Although details will be described later, in the present embodiment, the attention area of the user U is specified based on the voice of the user U picked up by the microphone 13.

[0019] The communication device 14 is hardware (a transmission / reception device) for communicating with the management device 30 via the communication network NW. The communication device 14 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 14 transmits the image data given from the processing device 18 to the management device 30. Further, the communication device 14 supplies the management table TBL received from the management device 30 to the processing device 18. Note that the communication device 14 may communicate with the management device 30 without going through the communication network NW.

[0020] The communication device 15 is hardware (transmitting and receiving device) for communicating with the glasses-type display device 20 via a wired connection. The communication device 15 supplies image data received from the glasses-type display device 20 to the processing device 18. The communication device 15 also transmits image data received from the processing device 18 to the glasses-type display device 20. The communication device 15 may also communicate with the glasses-type display device 20 wirelessly.

[0021] The storage device 17 is a recording medium that can be read by the processing device 18. The storage device 17 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory). The storage device 17 pre-stores a program PR1 that causes the processing device 18 to execute the method for identifying the area of ​​interest of this disclosure. In addition, the processing device 18 writes the management table TBL received from the management device 30 to the storage device 17.

[0022] The processing unit 18 includes one or more CPUs (Central Processing Units). One or more CPUs are an example of one or more processors. Each processor and CPU is an example of a computer. The processing unit 18 reads program PR1 from the storage device 17. Operating according to program PR1, the processing unit 18 transmits image data received from the glasses-type display device 20 using the communication device 15 to the management device 30 using the communication device 14. Also, the processing unit 18 operating according to program PR1 writes the management table TBL received from the management device 30 using the communication device 14 to the storage device 17.

[0023] Furthermore, the processing unit 18, which operates according to program PR1, functions as the speech recognition unit 181, the identification unit 182, and the display control unit 183 shown in Figure 4. In other words, the speech recognition unit 181, the identification unit 182, and the display control unit 183 in Figure 4 are software modules realized by operating the processing unit 18 according to the software.

[0024] The speech recognition unit 181 converts the sound represented by the sound data generated by the microphone 13 into a string of characters. That is, the speech recognition unit 181 performs speech recognition on the user U's voice according to a predetermined speech recognition algorithm. Existing technologies may be used as appropriate for the speech recognition algorithm. The speech recognition unit 181 generates recognition string data that represents the result of speech recognition on the user U's voice, i.e., a string of one or more words spoken by the user.

[0025] The identification unit 182 identifies the area of ​​interest based on the recognition result from the speech recognition unit 181. More specifically, the identification unit 182 refers to the management table TBL to determine whether the string represented by the recognition string data generated by the speech recognition unit 181 matches any of the identification information stored in the management table TBL. If the string represented by the recognition string data matches any of the identification information, the identification unit 182 identifies the area indicated by the area information corresponding to the identification information that matches the string represented by the recognition string data as the area of ​​interest.

[0026] The display control unit 183 displays a virtual object corresponding to the area of ​​interest identified by the identification unit 182 in the display area of ​​the glasses-type display device 20 that overlaps with the area of ​​interest identified by the identification unit 182. More specifically, the display control unit 183 generates image data representing an image in which the image representing the virtual object information corresponding to the identification information that matches the string represented by the recognition string data is placed in the area indicated by the area information corresponding to the identification information. The display control unit 183 then transmits this image data to the glasses-type display device 20 using the communication device 15, thereby displaying the image represented by the image data on the glasses-type display device 20.

[0027] Furthermore, the processing unit 18, which operates according to program PR1, executes the display method shown in Figure 5 each time sound data is output from the microphone 13. As shown in Figure 5, this display method includes the processes of steps SA110 to SA140.

[0028] In step SA110, the processing unit 18 functions as a speech recognition unit 181. In step SA110, the processing unit 18 generates recognized string data by performing speech recognition on the sound represented by the sound data output from the microphone 13.

[0029] In steps SA120 and SA130, the processing unit 18 functions as a specific unit 182. In step SA120, the processing unit 18 determines whether the string represented by the recognition string data generated in step SA110 matches any of the identification information stored in the management table TBL by referring to the management table TBL.

[0030] If the result of step SA120 is "Yes," that is, if the string represented by the recognition string data matches any of the identification information, the processing unit 18 executes the process in step SA130. In step SA130, the processing unit 18 identifies the area indicated by the area information stored in the management table TBL as the area of ​​interest, in association with the identification information that matches the string represented by the recognition string data generated in step SA110. If the result of step SA120 is "No," that is, if the string represented by the recognition string data does not match any of the identification information, the processing unit 18 terminates this display method without executing the processes after step SA130.

[0031] In step SA140, which follows step SA130, the processing unit 18 functions as a display control unit 183. In step SA140, the processing unit 18 retrieves virtual object information stored in the management table TBL, associating it with identification information that matches the string represented by the recognition string data generated in step SA110, and generates image data representing an image in which the virtual object represented by the virtual object information is placed in the area of ​​interest identified in step SA130. The processing unit 18 then supplies the generated image data to the glasses-type display device 20, thereby displaying the image represented by the image data on the glasses-type display device 20.

[0032] Figure 6 is a block diagram showing an example configuration of a glasses-type display device 20. The glasses-type display device 20 includes a display unit 2a, a communication device 2b, an imaging device 2c, a storage device 2d, a processing device 2e, and a bus 2f. The display unit 2a, the communication device 2b, the imaging device 2c, the storage device 2d, and the processing device 2e are interconnected by a bus 2f for communicating information. The bus 2f may be configured using a single bus, or different buses may be configured between elements such as devices.

[0033] The display unit 2a is a transmissive display unit that transmits light. Light representing real space passes through the display unit 2a. The display unit 2a displays an image of a virtual object under the control of the processing unit 2e. When user U wears the glasses-type display device 20, the display unit 2a is positioned in front of user U's left and right eyes. User U, wearing the glasses-type display device 20, sees both the real space represented by the light transmitted through the display unit 2a and the image of the virtual object displayed on the display unit 2a.

[0034] More specifically, the display unit 2a includes a lens for the left eye, a display panel for the left eye, an optical element for the left eye, a lens for the right eye, a display panel for the right eye, and an optical element for the right eye. The display panel for the left eye and the display panel for the right eye are, for example, liquid crystal panels or organic EL (Electro-Luminescence) panels. The display panel for the left eye displays an image represented by image data provided by the processing device e. The optical element for the left eye is an optical element that guides light emitted from the display panel for the left eye to the lens for the left eye. Similarly, the display panel for the right eye displays an image represented by image data provided by the processing device 2e. The optical element for the right eye is an optical element that guides light emitted from the display panel for the right eye to the lens for the right eye.

[0035] Each of the lenses, one for the left eye and one for the right eye, has a half-mirror. The half-mirror in the left eye lens transmits light representing real space, guiding it to the user U's left eye. The half-mirror in the left eye lens also reflects the light guided by the left eye's optical component back to the user U's left eye. The half-mirror in the right eye lens transmits light representing real space, guiding it to the user U's right eye. The half-mirror in the right eye lens reflects the light guided by the right eye's optical component back to the user U's right eye.

[0036] Communication device 2b is hardware (transceiver / receiver device) for communicating with the mobile device 10 via a wired connection. Communication device 2b may also communicate with the mobile device 10 wirelessly.

[0037] The spectacle-type display device 20 has a spectacle-type frame that supports lenses for the left eye and the right eye, and the imaging device 2c (e.g., a camera) is mounted on the bridge of the frame. Under the control of the processing device 2e, the imaging device 2c captures an image of real space seen through the spectacle-type display device 20 by the user U wearing the spectacle-type display device 20, that is, an image of the user U's field of view. The imaging device 2c outputs image data representing the captured image to the processing device 2e.

[0038] The storage device 2d is a recording medium readable by the processing device 2e. Like the storage device 17, the storage device 2d includes non-volatile memory and volatile memory. The storage device 2d stores program PR2. The processing device 2e includes one or more CPUs. The processing device 2e reads program PR2 from the storage device 2d. The processing device 2e functions as the operation control unit 2e1 by executing program PR2.

[0039] The motion control unit 2e1 controls the operation of the glasses-type display device 20. The motion control unit 2e1 transmits image data output from the imaging device 2c to the portable device 10 using the communication device 2b. The motion control unit 2e1 also supplies image data received from the portable device 10 via the communication device 2b to the display unit 2a. The display unit 2a displays the image represented by the image data supplied by the motion control unit 2e1. As described above, the image represented by the image data transmitted by the portable device 10 is an image in which a virtual object is placed in the area of ​​interest within the user U's field of view. Since this image is displayed on the display unit 2a, the user U sees an image of real space in which the virtual object is superimposed on the area of ​​interest within the user U's field of view.

[0040] For example, suppose user U, wearing glasses-type display device 20 on their head and a portable device 10 on their body, is looking at poster P shown in Figure 2 through the glasses-type display device 20, and the storage device 17 of the portable device 10 has the management table TBL shown in Figure 3 stored in it. Figure 7 shows an example of the field of view A that user U sees through the glasses-type display device 20. If the English word that user U is focusing on while looking at field of view A shown in Figure 7 is "Patients", user U will pronounce it as "payshentz".

[0041] User U's voice is picked up by the microphone 13 of the portable device 10, and based on the recognition result by the voice recognition unit 181 of this voice, the area corresponding to "Patients" is identified as the area of ​​interest. Image data representing an image in which a virtual object corresponding to "Patients" (the Japanese translation of "Patients") is placed in this area of ​​interest is transmitted from the portable device 10 to the glasses-type display device 20. As a result of the image represented by this image data being displayed on the display unit 2a, as shown in Figure 8, User U's eyes see a real-space image in which the virtual object VOB is superimposed on the area of ​​interest in the field of view A.

[0042] As described above, according to this embodiment, the user U can specify a region of interest that includes a real object that the user U is focusing on, and a virtual object can be displayed superimposed on the region of interest specified by the user U.

[0043] In addition, in this embodiment, user U can specify the area of ​​interest by voice, and does not need to operate the mouse or keyboard when specifying the area of ​​interest. Since it is not necessary to operate the mouse or keyboard when specifying the area of ​​interest, convenience is not reduced. Thus, according to this embodiment, it is possible to allow user U to specify the area of ​​interest in their field of view and display a virtual object in the area of ​​interest specified by user U without reducing user convenience.

[0044] B: Transformation This disclosure is not limited to the embodiments illustrated above. Specific variations are as follows. Two or more embodiments arbitrarily selected from the following examples may be combined. B-1: Torture 1 To prevent the user U's attention from being diverted from the area of ​​interest, the display control unit 183 may display a semi-transparent image on the portion of the display area of ​​the display unit 2a other than the area of ​​interest. When a semi-transparent image is displayed on the portion of the display area of ​​the display unit 2a other than the area of ​​interest, some of the light that passes through the portion other than the area of ​​interest from real space and enters the user U's eyes is blocked. As a result, as shown in Figure 9, the portion of the user's field of view other than the area of ​​interest seen through the glasses-type display device 20 becomes blurred, improving the user U's sense of immersion. In Figure 9, the blurring of the portion other than the area of ​​interest is represented by diagonal hatching. The glasses-type display device 20 may be configured to allow partial control of the transmittance of the left-eye lens and the right-eye lens. In this case, blocking of light that passes through the portion other than the area of ​​interest may be achieved by controlling the transmittance.

[0045] B-2: Modification 2 The identification information may be a series of numbers assigned one by one to each of the multiple physical objects. In this case, the display control unit 183 may display the identification information on the glasses-type display device 20 in association with each of the multiple physical objects, as shown in Figure 10. In the example shown in Figure 10, a series of numbers (circled numbers in the illustrated example) are assigned to each of the multiple physical objects. When the identification information is a series of numbers, the user U specifies the area of ​​interest by uttering the number assigned to the physical object of interest. For example, if the English word of interest is "Department," the number assigned to "Department" is "3," so the user U specifies the area of ​​interest by uttering "san" in Japanese or "three" in English. In this case, when the identification information (number) corresponding to any of the multiple physical objects is recognized by the speech recognition unit 181, the identification unit 182 identifies the area of ​​interest based on the recognized number.

[0046] B-3: Modification 3 The display control unit 183 may display a virtual object on the glasses-type display device 20 to instruct user U to launch a dictionary application (for example, an English-Japanese dictionary application). In this case, the processing unit 18 launches the English-Japanese dictionary application when sound data indicating the virtual object is provided from the microphone 13. Furthermore, the voice used to specify the target English word is not limited to a voice pronouncing the English word, but may also be a voice reading out the letters that make up the English word in the order they appear in the English word. For example, if the target English word is "World," user U specifies the area of ​​interest by uttering "W O R L D." In this case, the identification information may be a string of characters in which the pronunciations of the letters that make up the English word are arranged in the order they appear in the English word. Note that the actual object in this disclosure is not limited to an English word, but may also be a word in another language such as French, German, or Chinese. Furthermore, the location of the actual object in this disclosure is not limited to a poster, but may also be a magazine, book, or newspaper, or a menu in a restaurant, an instruction manual for equipment, various documents, or a sign.

[0047] B-4: Modification 4 The glasses-type display device 20 may be equipped with a detection device that detects the user U's gaze. In this case, the identification unit 182 may roughly identify the area occupied by the area of ​​interest in the user U's field of view based on the gaze detected by the detection device, and the display control unit 183 may perform display control to emphasize that area. For example, the display control unit 183 may display an enlarged image of the area occupied by the area of ​​interest on the display unit 2a, or it may display an image of a frame surrounding that area on the display unit 2a. Furthermore, the glasses-type display device 20 may be configured so that the user U can specify a region of interest by touching a portion of the field of view A in Figure 7 with their fingertip. In this case, the display control unit 183 may detect the region specified by the user U based on the image captured by the imaging device 2c and highlight the detected region as a candidate for translation.

[0048] B-5: Modification 5 In the above embodiment, program PR1 was stored in the storage device 17 of the portable device 10, but program PR1 may be manufactured or sold as a standalone product. When selling program PR1, the provider of program PR1 may write program PR1 to a computer-readable recording medium such as flash ROM and distribute it, or it may be distributed by download via a telecommunications line.

[0049] B-6: Torture 6 In the above embodiment, the speech recognition unit 181, the identification unit 182, and the display control unit 183 were software modules. However, one, more, or all of the speech recognition unit 181, the identification unit 182, and the display control unit 183 may be hardware modules. Specific examples of hardware modules include DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), etc.

[0050] B-7: Modification 7 In the above embodiment, the portable device 10 has a voice recognition unit 181, a specific unit 182, and a display control unit 183. However, the glasses-type display device 20 or the management device 30 may also have a voice recognition unit 181, a specific unit 182, and a display control unit 183. The voice recognition unit 181, the specific unit 182, and the display control unit 183 may be distributed and provided in any two or all of the glasses-type display device 20, the portable device 10, and the management device 30. Since the voice recognition processing by the voice recognition unit 181 is a high-load process, it is preferable that the voice recognition unit 181 be provided in the management device 30 or the portable device 10 rather than the glasses-type display device 20. Furthermore, it is preferable that the voice recognition unit 181 be provided in the management device 30 rather than the portable device 10.

[0051] C: Other (1) In the embodiments described above, ROM and RAM were given as examples of storage devices 17 and 2d, but storage devices 17 and 2d may be flexible disks, magneto-optical disks (e.g., compact disks, digital multipurpose disks, Blu-ray® disks), smart cards, flash memory devices (e.g., cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy® disks, magnetic strips, databases, servers, or other suitable storage media.

[0052] (2) In the embodiments described above, the information, signals, etc. may be represented using any of the various different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0053] (3) In the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.

[0054] (4) In the embodiments described above, the determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0055] (5) The processing procedures, sequences, flowcharts, etc., exemplified in the embodiments described above may be reordered, as long as they do not contradict each other. For example, in the methods described herein, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.

[0056] (6) Each function illustrated in Figure 4 is implemented by any combination of at least one of hardware and software. Furthermore, the method of implementing each function block is not particularly limited. That is, each function block may be implemented using one device that is physically or logically coupled, or it may be implemented using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired, wireless, etc.). A function block may also be implemented by combining the above one device or the above multiple devices with software.

[0057] (7) In the embodiments described above, the program should be interpreted broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether the software is called software, firmware, middleware, microcode, hardware description language or by any other name.

[0058] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0059] (8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.

[0060] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a given value, or other corresponding information.

[0061] (10) In the embodiments described above, the portable device may be a Mobile Station (MS). A Mobile Station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms. In this disclosure, terms such as “Mobile Station,” “User Terminal,” “User Equipment (UE),” and “Terminal” may be used interchangeably.

[0062] (11) In the embodiments described above, the terms “connected,” “coupled,” or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.

[0063] (12) In the embodiments described above, the phrase “based on” does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”

[0064] (13) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or other data structure), ascertaining, etc. “Determining” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," "considering," etc.

[0065] (14) Where the terms “include,” “including,” and variations thereof are used in the embodiments described above, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.

[0066] (15) In the present disclosure, if articles are added by translation, such as a, an, and the in English, the present disclosure may include the fact that the noun following these articles is plural.

[0067] (16) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combine” may be interpreted in the same way as “different.”

[0068] (17) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed in practice. Furthermore, notification of certain information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).

[0069] D: Modes understood from the above-described forms or modified forms Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are for illustrative purposes only and are not intended to be restrictive in any way. The following forms can be understood from at least one of the embodiments or modifications described above.

[0070] A glasses-type display device according to the first embodiment, which has a transparent display unit 2a on which virtual objects are displayed, may also include a identification unit 182 and a display control unit 183. The identification unit 182 identifies an area of ​​interest in the user U's field of view that includes a real object that the user U is fixated on, based on the voice spoken by the user U. The display control unit 183 causes a virtual object corresponding to the real object to be displayed in the display area of ​​the display unit 2a that overlaps with the area of ​​interest in the user U's field of view. According to the glasses-type display device according to the first embodiment, the user U can specify the real object to be fixated on by voice, so a virtual object can be displayed in the specified area of ​​interest without impairing the user U's convenience.

[0071] In the second embodiment (an example of the first embodiment), the field of view of user U may include multiple real objects. Also, in the second embodiment, one real object that user U is fixated on may be specified from multiple real objects by the voice spoken by user U. The glasses-type display device according to the second embodiment may further include a voice recognition unit 181 that recognizes the voice spoken by user U. The identification unit 182 in the glasses-type display device according to the second embodiment may identify the area of ​​interest based on the recognition result of the voice recognition unit 181. The glasses-type display device according to the second embodiment can identify the area of ​​interest based on the recognition result of the voice recognition unit 181 in response to user U's voice.

[0072] In the glasses-type display device according to the third embodiment (an example of the second embodiment), the display control unit 183 may display identification information that uniquely identifies each of the multiple physical objects in the display area of ​​the display unit 2a, associating it with each of the multiple physical objects. Furthermore, in the glasses-type display device according to the third embodiment (an example of the second embodiment), the identification unit 182 may identify a region of interest based on the identification information recognized by the voice recognition unit 181 when the voice recognition unit 181 recognizes the identification information corresponding to any of the multiple physical objects. The glasses-type display device according to the third embodiment can identify a region of interest based on the voice of user U indicating any of the multiple pieces of identification information displayed in the display area of ​​the display unit 2a, associating it with each of the multiple physical objects.

[0073] The fourth embodiment of the display system includes a glasses-type display device worn on the head of user U, equipped with a transparent display unit 2a on which virtual objects are displayed, a identification unit 182, and a display control unit 183. The identification unit 182 identifies an area of ​​interest in the user U's field of view that includes a real object that user U is fixated on, based on the voice spoken by user U. The display control unit 183 displays a virtual object corresponding to the real object in the display area of ​​the display unit 2a that overlaps with the area of ​​interest in the user U's field of view. According to the fourth embodiment of the display system, user U can specify the real object to be fixed on by voice, so that a virtual object can be displayed in the specified area of ​​interest without impairing user U's convenience. [Explanation of symbols]

[0074] 1...Display system, 10...Portable device, 20...Eyeglass-type display device, 11...Input device, 12...Output device, 13...Microphone, 14,15,2b...Communication device, 17,2d...Storage device, 18,2e...Processing device, 181...Speech recognition unit, 182...Identification unit, 183...Display control unit, 19,2f...Bus, 2a...Display unit, 2e1...Operation control unit, PR1,PR2...Program.

Claims

1. A spectacle-type display device having a transparent display unit on which virtual objects are displayed, An identification unit identifies an area of ​​interest, including a real object that the user is fixated on, within the user's field of vision, based on the voice spoken by the user. The display unit includes a display control unit that displays a first virtual object, which is a virtual object corresponding to the actual object, in a display area that overlaps with the area of ​​interest in the field of view. The aforementioned actual object is a word in the first language, The first virtual object is a translation of the word in a second language different from the first language, The display control unit causes the display unit to display a second virtual object, which is a virtual object that instructs the user to launch a dictionary application that associates the word with its translation in a second language. Glasses type display device.

2. The aforementioned real object is a single real object, The field of view includes a plurality of real objects, including the one real object, The aforementioned audio indicates that one of the multiple physical objects the user is focusing on is specified. The system further includes a voice recognition unit that recognizes the aforementioned voice, The identification unit identifies the area of ​​interest based on the recognition result of the speech recognition unit. The spectacle-type display device according to feature 1.

3. The display control unit displays identification information that uniquely identifies each of the plurality of physical objects in the display area, associating it with each of the plurality of physical objects. The identification unit, when identification information corresponding to any of the plurality of real objects is recognized by the speech recognition unit, identifies the area of ​​interest based on the identification information recognized by the speech recognition unit. The spectacle-type display device according to feature 2.

4. A glasses-type display device worn on the user's body, An identification unit identifies an area of ​​interest, including a real object that the user is fixated on within the user's field of vision, based on the voice spoken by the user. The spectacle-type display device includes a display control unit that causes a first virtual object, which is a virtual object corresponding to the real object, to be displayed in the display area of ​​the spectacle-type display device that overlaps with the area of ​​focus of the field of view, The aforementioned actual object is a word in the first language, The first virtual object is a translation of the word in a second language different from the first language, The display control unit causes the glasses-type display device to display a second virtual object, which is a virtual object that instructs the user to launch a dictionary application that associates the word with its translation in a second language. A display system equipped with the following features.

Images