Information display system, information display method, and program

Abstract

To cause a user to appropriately visually recognize information related to a target object to which the user is not paying close attention in his or her surroundings, using a wearable terminal worn by the user.SOLUTION: An information display system includes: an acquisition unit for acquiring detection information including information on a posture of a user and information on a line of sight of the user; a projection unit for projecting an image onto a projection target located in a predetermined projection direction with respect to the user; and a projection control unit for controlling the image to be projected onto the projection target based on the detection information and information on the projection direction.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an information display system, an information display method, and a program.

Background Art

[0002] Techniques have been developed to provide information about an object in front of a user using a wearable terminal worn by the user.

[0003] For example, in a glasses-type terminal worn on the user's head, display data is specified and displayed from 3D data corresponding to a captured image and position information, and based on the user's line-of-sight direction estimated from sensor information (see, for example, Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the technique disclosed in Patent Document 1, for example, various types of information about an object on which a user is focusing can be superimposed and displayed.

[0005] However, in the conventional technique, the wearable terminal has been limited to a usage method of displaying more information for objects within the range of the user's viewing angle. For example, in a glasses-type terminal, since the area where information can be displayed is limited to the lens surface, even if there is an object with a higher degree of importance other than the object on which the user is focusing around the user, the information may not be transmitted to the user.

[0006] One embodiment of the present invention has been made in view of the above problems, and appropriately allows a user to visually recognize information about surrounding objects that the user is not gazing at, using a wearable terminal worn by the user.

Means for Solving the Problems

[0007] To solve the above problems, an information display system according to one embodiment of the present invention is An information display system that uses a wearable device worn by the user to allow the user to see information about surrounding objects that the user is not looking at, the An acquisition unit that acquires detection information including information related to the user's gaze, and a projection target that is in a predetermined projection direction relative to the user. to A projection unit that projects an image, and the detection information to Based on the above, The projection part It comprises a projection control unit that controls the image to be projected, and the projection control unit is Based on the detection information, the user's effective field of view, stable field of gaze, and peripheral field of view are estimated, and if the object is outside the peripheral field of view, the projection unit is controlled to project an image of the object onto a projection target that is within the effective field of view, the stable field of gaze, or the peripheral field of view. [Effects of the Invention]

[0008] According to one embodiment of the present invention, a wearable device worn by the user can be used to appropriately display information about surrounding objects that the user is not looking at. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the outline of an information display system in one embodiment. [Figure 2] Figure (1) shows an example of the system configuration of an information display system according to one embodiment. [Figure 3] Figure (2) shows an example of the system configuration of an information display system according to one embodiment. [Figure 4] This figure shows an example of the hardware configuration of a first wearable terminal according to one embodiment. [Figure 5] This figure shows an example of the hardware configuration of a second wearable terminal according to one embodiment. [Figure 6] This figure shows an example of the hardware configuration of a wearable device according to one embodiment. [Figure 7] This figure shows an example of a computer hardware configuration according to one embodiment. [Figure 8] Figure (1) shows an example of the functional configuration of an information display system according to one embodiment. [Figure 9] Figure (2) shows an example of the functional configuration of an information display system according to one embodiment. [Figure 10] It is a diagram (3) showing an example of the functional configuration of an information display system according to an embodiment. [Figure 11] It is a diagram (4) showing an example of the functional configuration of an information display system according to an embodiment. [Figure 12] It is a diagram (5) showing an example of the functional configuration of an information display system according to an embodiment. [Figure 13] It is a diagram showing an example of the functional configuration of a projection control unit according to an embodiment. [Figure 14] It is a flowchart showing an example of the processing of an information display system according to an embodiment. [Figure 15] It is a diagram for explaining a visual field area according to an embodiment. [Figure 16] It is a flowchart showing an example of the determination process of a projection method according to the first embodiment. [Figure 17] It is a diagram showing an image example of a normal display method according to the first embodiment. [Figure 18] It is a diagram showing an image example of a first highlighting display according to the first embodiment. [Figure 19] It is a diagram showing an image example of a second highlighting display according to the first embodiment. [Figure 20] It is a flowchart showing an example of the determination process of a projection method according to the second embodiment. [Figure 21] It is a flowchart showing an image example of a display method according to the second embodiment. [Figure 22] It is a flowchart showing an example of the processing of an information display system according to the third embodiment. [Figure 23] It is a diagram showing an image example of a display method according to the third embodiment. [Figure 24] It is a flowchart showing an example of the determination process of a projection target according to the fourth embodiment. [Figure 25] It is a diagram showing an example of a display method according to the fourth embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, each embodiment of the present invention will be described with reference to the attached drawings.

[0011] <Overview> Figure 1 is a diagram illustrating the overview of an information display system according to one embodiment. The information display system 1 includes, as an example, a first wearable terminal 100-1 worn by a user 2 on their head or the like, and a second wearable terminal 100-2 worn on the torso, such as the chest or shoulder. The first wearable terminal 100-1 and the second wearable terminal 100-2 are connected to each other so as to be able to communicate with each other via wireless or wired communication.

[0012] The first wearable terminal 100-1 is equipped with, for example, one or more peripheral cameras and has the function of acquiring images of the user's surroundings. The first wearable terminal 100-1 is also equipped with an IMU (Inertial Measurement Unit) or an EOG (Electro-Oculogram) sensor and has the function of acquiring detection information for estimating the user's gaze.

[0013] The second wearable terminal 100-2, for example, is equipped with one or more projectors and has the function of projecting an image 4 onto a projection target 3 that is in a predetermined projection direction relative to the user 2. The second wearable terminal 100-2 is also equipped with a 3D sensor such as LiDAR (Light Detection And Ranging) that acquires 3D point cloud data of the area around the user 2, and a wearer camera that captures images of the user wearing the first wearable terminal 100-1.

[0014] The information display system 1 acquires detection information, including information about the user's posture or gaze, detected by, for example, the first wearable terminal 100-1 or the second wearable terminal 100-2. Furthermore, when projecting an image 4 onto a projection target 3 located in a predetermined projection direction for the user, the information display system 1 controls the projection method of the image 4 to be projected onto the projection target 3 based on the acquired detection information and the projection direction information.

[0015] As a specific example, the information display system 1 estimates the user's gaze direction from the acquired detection information, and if the difference between the user's gaze direction and the projection direction is greater than or equal to a threshold, it highlights the image 4 to be projected onto the projection target 3 (for example, by flashing, vibrating, or inverting).

[0016] As another example, if the difference between the user's (user's) line of sight and the projection direction is even greater, the information display system 1 will display an image indicating the position or direction of the projection target on another projection target within the user's (user's) field of view.

[0017] Thus, according to the information display system 1 of this embodiment, information about surrounding objects that the user 2 is not looking at is appropriately displayed to the user using a wearable terminal worn by the user 2.

[0018] <System Configuration> Figures 2 and 3 show an example configuration of an information display system according to one embodiment. In the example in Figure 2(A), the information display system 1 has a first wearable terminal 100-1 and a second wearable terminal 100-2, as described in Figure 1, connected to each other via wireless or wired communication.

[0019] However, this is just one example, and the first wearable terminal 100-1 and the second wearable terminal 100-2 may be composed of a single wearable terminal 100, as shown in Figure 2(A). In this case, the wearable terminal 100 may be attached to the head or other part of the user 2, similar to the first wearable terminal 100-1, or it may be attached to the torso or other part of the user 2, similar to the second wearable terminal 100-2.

[0020] Furthermore, as shown in Figure 3, the information display system 1 may include an information processing device 300 that is communicably connected to a first wearable terminal 100-1 and a second wearable terminal 100-2 via wireless or wired communication.

[0021] The information processing device 300 is an information terminal having the configuration of a computer, such as a PC (Personal Computer), a tablet terminal, or a smartphone. The information processing device 300 controls the first wearable terminal 100-1 and the second wearable terminal 100-2 by executing a predetermined program, thereby controlling the information display method according to this embodiment.

[0022] <Hardware Configuration> Next, we will describe an example of the hardware configuration of each device included in the information display system 1.

[0023] (First wearable device) Figure 4 shows an example of the hardware configuration of a first wearable terminal according to one embodiment. The first wearable terminal 200-1 includes, for example, a CPU (Central Processing Unit) 401, memory 402, storage device 403, communication I / F (Interface) 404, one or more peripheral cameras 405, IMU 406, EOG sensor 407, audio circuit 408, speaker 409, microphone 410, and bus 411.

[0024] The CPU 401 is a processing unit that realizes the various functions of the first wearable terminal 100-1 by reading programs and data stored in a storage medium such as the storage device 403 onto the memory 302 and executing processing. The memory 402 includes, for example, RAM (Random Access Memory) used as the work area of ​​the CPU 401, and ROM (Read Only Memory) that stores the program for starting the CPU 401. The storage device 403 is a large-capacity non-volatile storage device that stores the OS (Operating System), applications, and various data, and is implemented by, for example, an SSD (Solid State Drive) or HDD.

[0025] The communication interface 304 is a communication interface for communicating with the second wearable terminal 100-2, such as short-range wireless communication, wireless LAN (Local Area Network), or wired communication. The communication interface 304 also includes a communication interface such as LAN or WAN (Wide Area Network) for connecting to communication networks such as the Internet and LAN.

[0026] One or more peripheral cameras 405 are imaging devices that capture images (videos) of the area around user 2. A peripheral camera 405 may be a 360-degree camera capable of capturing images (videos) in all directions with a single unit, or it may be composed of multiple cameras. The information display system 1 can, for example, perform Visual SLAM (Simultaneous Localization and Mapping) to estimate its own position and create an environmental map using images captured by one or more peripheral cameras 405.

[0027] The IMU406 is an inertial measurement device that acquires, for example, posture information of user 2. For example, the IMU406 detects three-dimensional angular velocity and acceleration using a three-axis gyro sensor and a three-directional accelerometer. The EOG sensor 407 is an electrooculography sensor for estimating the gaze of user 2. The first wearable terminal 100-1 only needs to have at least one of the IMU406 and the EOG sensor 407.

[0028] The audio circuit 408 outputs an audio signal to the speaker 409 and amplifies the audio signal input from the microphone 410, according to the control from the CPU 401. The speaker 409 converts the audio signal input from the audio circuit 408 into audio and outputs it. The microphone 410 acquires ambient sound, converts it into an audio signal, and outputs it to the audio circuit 408. The first wearable terminal 200-1 does not necessarily have to have some or all of the audio circuit 408, speaker 409, and microphone 410. The bus 411 is connected in common to all of the above components and transmits, for example, address signals, data signals, and various control signals.

[0029] (Second wearable device) Figure 5 shows an example of the hardware configuration of a second wearable terminal according to one embodiment. The second wearable terminal 200-2 includes, for example, a CPU 501, memory 502, storage device 503, communication I / F 504, LiDAR 505, wearer camera 506, one or more projectors 507, and a bus 508. Of these, the CPU 501, memory 502, storage device 503, communication I / F 504, and bus 508 are the same as the CPU 401, memory 402, storage device 403, communication I / F 404, and bus 411 described in Figure 4, so their description is omitted here.

[0030] The LiDAR505 is a ranging device that measures the distance to an object by irradiating the object with light such as laser light and detecting the reflected light with an optical sensor. The LiDAR505 is mainly used to acquire information such as the distance to the projection surface of the projection target, its inclination, and its shape. However, it is not limited to this, and the information display system 1 may also use the LiDAR505 for SLAM (Simultaneous Localization and Mapping) to perform self-position estimation and environmental map creation.

[0031] The wearer camera 506 is a camera that photographs user 2. The information display system 1 can use the image of user 2 taken by the wearer camera 506 to estimate, for example, user 2's gaze direction. The information display system 1 may also use the image of user 2 taken by the wearer camera 506 to identify user 2.

[0032] One or more projectors 507 are image projection devices that project images 4, etc., onto the projection target 3. The information display system 1 may have multiple projectors 507, for example, when it is desired to project images 4, etc., over a wide area, when it is desired to project images 4, etc., onto multiple projection targets 3, or when it is desired to project images 4, etc., onto a projection target 3 located in a bright place. Here, the following explanation will be given assuming that there is only one projector 507.

[0033] (Wearable device) Figure 6 shows an example of the hardware configuration of a wearable terminal according to one embodiment. The wearable terminal 100 includes, for example, a CPU 601, memory 602, storage device 603, communication I / F 604, one or more peripheral cameras 605, IMU 606, EOG sensor 607, audio circuit 608, speaker 609, microphone 610, and bus 619. Note that each of the above components is the same as the CPU 401, memory 402, storage device 403, communication I / F 404, one or more peripheral cameras 405, IMU 406, EOG sensor 407, audio circuit 408, speaker 409, microphone 410, and bus 411 described in Figure 4, so their explanation is omitted here.

[0034] Furthermore, the wearable terminal 100 includes, for example, a LiDAR 611, a wearer camera 612, and one or more projectors. Note that each of the above components is the same as the LiDAR 505, wearer camera 506, and one or more projectors 507 described in Figure 5, so their explanation is omitted here.

[0035] As shown in Figure 6, the wearable terminal 100 has a configuration that integrates a first wearable terminal 100-1 and a second wearable terminal 100-2.

[0036] (Information processing device) The information processing device 300 has, for example, the hardware configuration of a computer 700 as shown in Figure 7. Alternatively, the information processing device 300 is composed of multiple computers 700.

[0037] Figure 7 shows an example of the hardware configuration of a computer according to one embodiment. The computer 700 includes, for example, a CPU 701, a ROM (Read Only Memory) 702, RAM 703, an HD (Hard Disk) 704, an HDD controller 705, a display 706, an external device connection I / F 707, a network I / F 708, a keyboard 709, a pointing device 710, a DVD-RW (Digital Versatile Disk Rewritable) drive 712, a media I / F 714, and a bus line 715, as shown in Figure 7.

[0038] Of these components, the CPU 701 controls the overall operation of the computer 700. The ROM 702 stores programs used to start the computer 700, such as the IPL (Initial Program Loader). The RAM 703 is used, for example, as the work area for the CPU 701. The HD 704 stores programs such as the OS (Operating System), applications, and device drivers, as well as various data. The HDD controller 705 controls the reading or writing of various data to the HD 704, for example, according to the control of the CPU 701.

[0039] Display 706 displays various information, such as a cursor, menu, window, text, or image. Note that Display 706 may be located outside of the computer 700. External device connection I / F 707 is an interface, such as USB (Universal Serial Bus), for connecting various external devices to the computer 700. Network I / F 708 is an interface for communicating with other devices, such as using a communication network.

[0040] The keyboard 709 is a type of input device equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 710 is a type of input device for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. Note that the keyboard 709 and the pointing device 710 may be located outside the computer 700. The DVD-RW drive 712 controls the reading or writing of various data to the DVD-RW 711, which is an example of a removable storage medium. Note that the DVD-RW 711 is not limited to DVD-RW; it may be other storage media.

[0041] The media interface 714 controls the reading or writing (storage) of data to or from the media 713, such as flash memory. The bus line 715 includes an address bus, a data bus, and various control signals for electrically connecting the above components.

[0042] Note that the hardware configuration of computer 700 shown in Figure 7 is just one example. Computer 700 can have any other configuration as long as it has, for example, a CPU 701, ROM 702, RAM 703, network I / F 708, and bus line 216.

[0043] <Functional Configuration> Figures 8 to 12 show examples of the functional configuration of an information display system according to one embodiment.

[0044] Figure 8 shows an example of the functional configuration of an information display system 1 according to one embodiment.

[0045] (Functional configuration of the first wearable device) The first wearable terminal 100-1 implements a functional configuration such as a communication unit 801, an acquisition unit 802, a projection control unit 803, a storage unit 804, and an information acquisition unit 805 by executing a predetermined program on the CPU 401, for example. At least some of the above functional configurations may be implemented by hardware.

[0046] The communication unit 801 performs communication processing to communicate with the second wearable terminal 100-2, for example, using the communication interface 404. Preferably, the communication unit 801 can also connect to the communication network N using the communication interface 404 and communicate with external devices such as an external server 810.

[0047] The acquisition unit 802 performs an acquisition process to acquire detection information, including information about the user 2's posture (for example, data output from the IMU 406) or information about the user 2's gaze (for example, data detected by the EOG sensor 407). The acquisition unit 802 may acquire information about the user 2's gaze from the second wearable terminal 100-2.

[0048] The projection control unit 803 executes projection control processing to control the image 4 to be projected onto the projection target 3, based on the detection information acquired by the acquisition unit 802 and information regarding the projection direction, which is the direction of the projection target 3 relative to the user 2. The specific functional configuration and processing details of the projection control unit 803 will be described later.

[0049] The memory unit 804 is implemented by, for example, a program executed by the CPU 401, a storage device 403, and memory 402, and stores data for identifying the projection target 3, and several thresholds, which will be described later. The data for identifying the projection target 3 may be, for example, 3D CAD (Computer Aided Design) data of the projection target 3 that has been stored in advance, or 3D data of the digital twin 811 acquired from an external server 810. Here, the digital twin 811 is a technology that reproduces a copy (twin) of the physical space in the digital space based on 3D information acquired from the physical space. For example, the 3D data acquired from the digital twin 811 includes various information or data such as 3D data of various objects around the user 2, information about the position of the projection target 3, and the image 4 to be projected onto the projection target.

[0050] Furthermore, the data used to identify the projection target 3 may be, for example, a machine learning model that has been pre-trained to identify whether or not an object included in an image captured by the surrounding camera 405 is the projection target 3.

[0051] The information acquisition unit 805 acquires data to be stored in the storage unit 804 (for example, 3D data of the digital twin 811, or a trained machine learning model, etc.) from an external server 810 or the like, as needed, and executes information acquisition processing to store the data in the storage unit 804.

[0052] (Functional configuration of the second wearable device) The second wearable terminal 100-2 implements the communication unit 806 and the projection unit 807, etc., by executing a predetermined program on the CPU 501, for example. At least some of the above functional configurations may be implemented by hardware.

[0053] The communication unit 806 performs communication processing to communicate with the first wearable terminal 100-1, for example, using the communication interface 504. The communication unit 806 may also have the function of connecting to the communication network N and communicating with external devices such as an external server 810 using the communication interface 404.

[0054] The projection unit 807 performs a projection process to project an image 4 or the like onto the projection target 3 using the projector 507, in accordance with instructions from the projection control unit 803.

[0055] Note that the functional configuration of the information display system 1 shown in Figure 8 is just one example. For example, the functional configurations of the first wearable terminal 100-1 and the second wearable terminal 100-2 in Figure 8 only need to be included in the information display system 1 and may be provided in either device.

[0056] Figure 9 shows another example of the functional configuration of the information display system 1 according to one embodiment. As shown in Figure 9, the functional configuration of the acquisition unit 802, projection control unit 803, storage unit 804, and information acquisition unit 805 may be located in the second wearable terminal 100-2. In this case, the acquisition unit 802 acquires detection information, including the aforementioned information regarding the posture of user 2 or information regarding the gaze of user 2, from the first wearable terminal 100-1.

[0057] In this case, the first wearable terminal 100 has a detection unit 901. The detection unit 901 is implemented, for example, by a program executed by the CPU 401, and acquires detection information from the IMU 406 or EOG sensor 407, etc., and transmits the acquired detection information to the second wearable terminal 100-2 via the communication unit 801. The other functional configurations may be the same as those described in Figure 8.

[0058] Figure 10 shows another example of the functional configuration of the information display system 1 according to one embodiment. As shown in Figure 10, the functions of the projection control unit 803, the storage unit 804, and the information acquisition unit 805 may be distributed and provided on the first wearable terminal 100-1 and the second wearable terminal 100-2.

[0059] In the example shown in Figure 10, the projection control unit 803a of the first wearable terminal 100-1 works in cooperation with the projection control unit 803b of the second wearable terminal 100-2 to realize the functions of the projection control unit 803 described in Figure 8. In addition, the memory unit 804a of the first wearable terminal 100-1 stores the information used by the projection control unit 803a from the information stored in the memory unit 804 described in Figure 8, and the information acquisition unit 805a acquires the information to be stored in the memory unit 804a as needed.

[0060] Similarly, the projection control unit 803b of the second wearable terminal 100-2 works in cooperation with the projection control unit 803a of the first wearable terminal 100-1 to realize the functions of the projection control unit 803 as described in Figure 8. In addition, the memory unit 804b of the second wearable terminal 100-2 stores the information used by the projection control unit 803b from the information stored in the memory unit 804 as described in Figure 8, and the information acquisition unit 805b acquires the information to be stored in the memory unit 804b as needed.

[0061] Figure 11 shows another example of the functional configuration of an information display system 1 according to one embodiment. As shown in Figure 11, each functional configuration of the information display system 1 may be realized by an integrated wearable terminal 100. In the example in Figure 11, the wearable terminal 100 realizes the communication unit 801, acquisition unit 802, projection control unit 803, storage unit 804, information acquisition unit 805, and projection unit 807, etc., by executing a predetermined program on the CPU 601. Note that each of the above functional configurations is the same as the communication unit 801, acquisition unit 802, projection control unit 803, storage unit 804, information acquisition unit 805, and projection unit 807 described in Figure 8, so their explanation is omitted here.

[0062] Figure 12 shows another example of the functional configuration of the information display system 1 according to one embodiment. As shown in Figure 12, among the functional configurations of the information display system 1, the acquisition unit 802, projection control unit 803, storage unit 804, and information acquisition unit 805, etc., may be provided by the information processing device 300.

[0063] In the example shown in Figure 12, the information processing device 300 implements the communication unit 1201, acquisition unit 802, projection control unit 803, storage unit 804, and information acquisition unit 805, etc., by executing a predetermined program on the CPU 701. Note that the acquisition unit 802, projection control unit 803, storage unit 804, and information acquisition unit 805 are the same as those described in Figure 8, so their explanation is omitted here.

[0064] The communication unit 1201 performs communication processing to communicate with the first wearable terminal 100-1 and the second wearable terminal 100-2, for example, using the network interface 708. The communication unit 1201 also connects the information processing device 300 to the communication network N and can communicate with an external server 810, etc.

[0065] In the example shown in Figure 12, the first wearable device 100-1 has, for example, the functional configuration shown in Figure 9. In another example, the first wearable device 100-1 may have the functional configuration shown in Figure 10.

[0066] Furthermore, the second wearable terminal 100-2 may have a functional configuration as shown in Figure 8, for example. As another example, the second wearable terminal 100-2 may have a functional configuration as shown in Figure 10.

[0067] If the first wearable terminal 100-1 has the functional configuration shown in Figure 10, the projection control unit 803, storage unit 804, and information acquisition unit 805 of the information processing device 300 will work in cooperation with the first wearable terminal 100-1 to realize the functions described in Figure 8. Similarly, if the second wearable terminal 100-2 has the functional configuration shown in Figure 10, the projection control unit 803, storage unit 804, and information acquisition unit 805 of the information processing device 300 will work in cooperation with the second wearable terminal 100-2 to realize the functions described in Figure 8.

[0068] Furthermore, if the first wearable terminal 100-1 and the second wearable terminal 100-2 have the functional configurations shown in Figure 10, the information processing device 300 will work in cooperation with the first wearable terminal 100-1 and the second wearable terminal 100-2 to realize each function.

[0069] As described above, the functional configurations of the first wearable terminal 100-1 and the second wearable terminal 100-2, as explained in Figure 8, only need to be present in the information display system 1, and may be present in either device.

[0070] (Functional configuration of the projection control unit) Figure 13 shows an example of the functional configuration of a projection control unit according to one embodiment. The projection control unit 803 includes, for example, a line of sight estimation unit 1301, a field of view area estimation unit 1302, a projection target determination unit 1303, a projection method determination unit 1304, and a projection instruction unit 1305, as shown in Figure 13.

[0071] The gaze estimation unit 1301 performs gaze estimation processing to estimate the direction of the user's gaze based on the detection information acquired by the acquisition unit 802. For example, the gaze estimation unit 1301 may acquire the direction of the user's gaze from the detection information acquired from the EOG sensor 407, which acquires the direction of the gaze. Alternatively, the gaze estimation unit 1301 may measure which direction the face is facing from the detection information acquired from the IMU 406 and estimate the direction of the user's gaze from that direction. Furthermore, the gaze estimation unit 1301 may analyze the image of the user 2 taken by the wearer camera 506 to estimate the direction of the user's gaze.

[0072] The field of view estimation unit 1302 estimates the field of view of user 2, centering on the direction of the user's gaze estimated by the gaze estimation unit 1301, and extending around it.

[0073] The projection target determination unit 1303 determines (identifies) the projection target 3 located around user 2 based, for example, on an image of the user's surroundings captured by the surrounding camera 605 and information about the projection target 3 (for example, 3D CAD data) that has been previously stored in the storage unit 804.

[0074] As another example, the projection target determination unit 1303 may determine the projection target by obtaining 3D information about the user 2's surroundings from the image captured by the surrounding camera 605 using VSALM and comparing it with the 3D information obtained from the digital twin 811. The 3D data obtained from the digital twin 811 includes at least information indicating the position of the projection target 3 and information such as the image 4 to be projected onto the projection target 3.

[0075] The projection method determination unit 1304 changes the projection method (content, position, highlighting, etc.) of the image 4, etc., to be projected onto the projection object 3, based on the difference between the gaze direction of the user 2 estimated by the gaze estimation unit 1301 and the direction of the projection object 3 relative to the user (projection direction).

[0076] The projection instruction unit 1305 executes a projection instruction process that instructs the projection of the image 4 onto the projection target 3, etc., using the projection method determined by the projection method determination unit 1304.

[0077] <Processing flow> Next, the processing flow of the information display method according to this embodiment will be described.

[0078] (Processing of the information display system) Figure 14 is a flowchart illustrating an example of processing in an information display system according to one embodiment. This process shows an example of processing performed by an information display system 1 having, for example, one of the functional configurations shown in Figures 8 to 13.

[0079] In step S1401, the acquisition unit 802 acquires detection information including information about the user 2's posture or information about the user 2's gaze. Here, information about the user 2's posture includes, for example, data such as three-dimensional angular velocity and acceleration detected by the IMU 406. Information about the user's gaze includes, for example, data indicating the direction of the user's gaze detected by the EOG sensor 407, or images of the user 2 taken by the wearer camera 506, etc.

[0080] In step S1402, the projection control unit 803 estimates the user's gaze direction based on the detection information acquired by the acquisition unit 802. For example, the gaze estimation unit 1301 of the projection control unit 803 acquires the user's gaze direction from the detection information acquired from the EOG sensor 407.

[0081] In step S1403, the projection control unit 803 estimates the user's line of sight area. Figure 15 is a diagram illustrating the field of view area according to one embodiment. A person's field of view, for example, as shown in Figure 15, is centered on the person's line of sight direction 1511, with the entire field of view extending outwards from there. This field of view can be broadly divided into three areas.

[0082] The first region is the effective field of view 1512, and its range is, for example, as shown in Figures 15(B) and (C), an area of ​​±10 degrees vertically and ±10 degrees horizontally with respect to the line of sight 1511. The second region is the stable gaze field of view 1513, and its range is, for example, as shown in Figures 15(B) and (C), an area of ​​±10 degrees or more and less than 35 degrees vertically and ±10 degrees or more and less than 55 degrees horizontally with respect to the line of sight 1511. The third region is the peripheral field of view 1514, and its range is, for example, as shown in Figures 15(B) and (C), an area of ​​±35 degrees or more and 60 degrees vertically and 55 degrees or more and less than 110 degrees horizontally with respect to the line of sight 1511.

[0083] The effective field of view 1512 is the area closest to the direction a person is looking and fixating on, and information displayed within this area is easily noticed. Here, fixation refers to, for example, the state of staring intently at a point or narrow area in space for a certain period of time or longer.

[0084] As we move from the effective field of view (1512) to the stable field of fixation (1513), and then from the stable field of fixation (1513) to the peripheral field of view (1514), it becomes more difficult for people to consciously see things as they move away from their line of sight, and they become less likely to notice information displayed in these areas.

[0085] Thus, because each region of a person's field of vision has a different capacity to process information, by dividing the field of vision into multiple regions according to the direction from the line of sight and changing the display method for each region, it becomes possible to make a person aware of information that is far away from the direction of their line of sight. Here, the number of regions into which the field of vision is divided can be any number, and the projection method can be gradually changed according to the distance.

[0086] Here, as an example, we will explain the following assuming that the field of view estimation unit 1302 of the projection control unit 803 estimates three areas of user 2 in step S1403: the effective field of view 1512, the stable field of gaze 1513, and the peripheral field of view 1514.

[0087] In step S1404, the projection control unit 803 determines the position of the projection target 3. For example, the projection target determination unit 1303 of the projection control unit 803 may determine the position of the projection target 3 from an image of the user 2's surroundings captured by a peripheral camera, using 3D data of the projection target 3 previously stored in the storage unit 804, and employing known template matching techniques or the like.

[0088] Alternatively, the projection target determination unit 1303 may perform self-position estimation and create an environmental map using SLAM from captured images taken by the surrounding camera or 3D data acquired by LiDAR, and determine the position of the projection target 3 by correlating it with the data of the digital twin 811.

[0089] In step S1405, the projection control unit 803 determines the image projection method. For example, the projection method determination unit 1304 of the projection control unit 803 changes the projection method of the image to be projected onto the projection target based on the difference between the user's line of sight direction 1511 and the direction of the projection target relative to the line of sight direction 1511. The process of determining the projection method by the projection method determination unit 1304 will be described later with examples of several embodiments.

[0090] In step S1406, the projection control unit 803 projects the image using the determined projection method. For example, the projection instruction unit 1305 of the projection control unit 803 identifies the projection direction of the object using SLAM based on images captured by the surrounding cameras or 3D data acquired by LiDAR, and instructs the projection unit 807 to project the image in the identified projection direction.

[0091] <Process for determining the projection method> [First Embodiment] Figure 16 is a flowchart showing an example of the process for determining the projection method according to the first embodiment. This process shows, for example, an example of the process performed by the projection method determination unit 1304 of the projection control unit 803 in step S1405 of Figure 14.

[0092] In step S1601, the projection method determination unit 1304 calculates the difference X between the user's line of sight direction and the projection direction in which the object is located.

[0093] In step S1602, the projection method determination unit 1304 determines whether the calculated X is less than or equal to the first threshold. Here, as an example for explanation, the range in which X is less than or equal to the first threshold corresponds to the effective field of view 1512 explained in Figure 15, and the following explanation is given accordingly.

[0094] If X is less than or equal to the first threshold, the projection method determination unit 1304 proceeds to step S1603. On the other hand, if X is not less than or equal to the first threshold, the projection method determination unit 1304 proceeds to step S1604.

[0095] When the process moves to step S1603, the projection method determination unit 1304 decides to project the image onto the object using the normal display method. Here, the normal display method means displaying the image as is, without performing the highlighting described later.

[0096] Figure 17 shows an example of a typical display method according to the first embodiment. As shown in Figure 17, when the projection target 1700 is within the effective field of view 1512 (when X is less than or equal to the first threshold), the information display system 1 projects the image 1701 onto the projection target 1700 as is. The example in Figure 17 shows a case where the image 1701 projected onto the projection target 1700 is an image representing the string "Confirmation Required".

[0097] When moving from step S1602 to step S1604, the projection method determination unit 1304 determines whether the calculated X is below the second threshold. Here, as an example for explanation, the range in which X is below the second threshold corresponds to the range of the stable field of gaze 1513 explained in Figure 15, and the following explanation is given accordingly.

[0098] If X is less than or equal to the second threshold, the projection method determination unit 1304 proceeds to step S1605. On the other hand, if X is not less than or equal to the first threshold, the projection method determination unit 1304 proceeds to step S1606.

[0099] When the process moves to step S1605, the projection method determination unit 1304 decides to project the image onto the object with a first highlighting. Here, the first highlighting means, for example, performing one level of highlighting, such as flashing, enlargement, judgment, or movement, so that the projected image attracts the attention of user 2.

[0100] Figure 18 is a diagram showing an example of a first highlighting according to the first embodiment. As shown in Figure 18, when the projection target 1700 is within the stable gaze field 1513 (when X is greater than the first threshold and less than or equal to the second threshold), the information display system 1 may project an image 1801 that flashes (an example of the first highlighting) onto the projection target 1700. As a result, when the projection target 1700 is within the stable gaze field 1513, the information display system 1 can project the image 1801 with emphasis so that the user 2's gaze is directed towards the projection target 1700.

[0101] When moving from step S1604 to step S1606, the projection method determination unit 1304 decides to project the image onto the object with a second highlighting. Here, the second highlighting refers to performing a two-stage highlighting, such as flashing, enlargement, judgment, or movement, so that the projected image attracts the user's attention even more. However, it is not limited to these, and the second highlighting can be any display method that is more noticeable or eye-catching than the first highlighting.

[0102] Figure 19 shows an image of an example of a second highlighting according to the first embodiment. As shown in Figure 19, when the projection target 1700 is outside the stable gaze field 1513 (when X is greater than the second threshold), the information display system 1 may project an image 1901 onto the projection target 1700 that flashes and moves (an example of a second highlighting). This allows the information display system 1 to further emphasize and project the image 1801 so that the user 2's gaze is directed more toward the projection target 1700 when the projection target 1700 is outside the stable gaze field 1513.

[0103] [Second Embodiment] Figure 20 is a flowchart showing an example of the process for determining the projection method according to the second embodiment. This process shows another example of the process executed by the projection method determination unit 1304 of the projection control unit 803 in step S1405 of Figure 14. Of the processes shown in Figure 20, steps S1601 to S1606 are the same as the process for determining the projection method according to the first embodiment described in Figure 6, so their explanation is omitted here.

[0104] If, in step S1606, the projection method determination unit 1304 decides to project the image onto the object with a second level of highlighting, then in step S2001, the projection method determination unit 1304 determines whether X is less than or equal to a third threshold.

[0105] If X is less than or equal to the third threshold, the projection method determination unit 1304 terminates the process shown in Figure 20. On the other hand, if X is not less than or equal to the third threshold, the projection method determination unit 1304 moves the process to step S2002.

[0106] When the process moves to step S2002, the projection method determination unit 1304 determines to project an image indicating the position or direction of the projection target onto another object where X is less than or equal to a third threshold.

[0107] Figure 21 shows an image of an example of a display method according to the second embodiment. In the example in Figure 21, the projection target 1700 is outside the user 2's peripheral field of view 1514 (an example of a third threshold). In such a case, the projection method determination unit 1304 decides to project an image indicating the position or direction of the projection target 1700 onto another projection target 2100 that is within the user 2's peripheral field of view 1514. In the example in Figure 21, an image including an arrow (an example of a display element) 2101 indicating the direction of the projection target 1700 is projected onto the other projection target 2100.

[0108] Preferably, the information display system 1 projects an image 1901 onto the projection target 1700, which is flashing and moving (second highlighting), even when the projection target 1700 is outside the user 2's peripheral field of view 514. This allows the user 2 to easily see the projection target 1700 and the image 1901 when they move their gaze, etc. The third threshold is not limited to the peripheral field of view 1514, and may be changed by, for example, a set value.

[0109] [Third Embodiment] In the third embodiment, an example of the process in which the information display system 1 acquires eye-tracking speed information of user 2 and changes thresholds (for example, a first threshold, a second threshold, or a third threshold, etc.) based on the acquired eye-tracking speed information will be described.

[0110] Figure 22 is a flowchart showing an example of the processing of the information display system according to the third embodiment. Of the processing shown in Figure 22, steps S1401 and S1403 to S1406 are the same as the processing of the information display system according to the first embodiment described in Figure 4, so their explanation is omitted here.

[0111] In step S2201, the projection control unit 803 estimates the user 2's gaze direction and gaze speed based on the detection information acquired by the acquisition unit 802. For example, the projection control unit 803 acquires the user 2's gaze direction and gaze speed from the detection information acquired from the EOG sensor 407.

[0112] In step S2202, the projection control unit 803 changes the thresholds based on the estimated gaze velocity. For example, if the gaze velocity within a predetermined time is below a predetermined value, the projection control unit 803 determines that user 2 is fixated on something and their field of view is narrowed, and lowers the first threshold, the second threshold, and the third threshold.

[0113] Figure 23 is a diagram illustrating an example of a display method according to the third embodiment. This diagram shows an image of what happens when the second threshold 2301 and the third threshold 2302 are lowered (their values ​​are reduced) from the state shown in Figure 19, where the stable field of gaze 1513 is the second threshold and the peripheral field of view 1514 is the third threshold. In the example of Figure 23, by lowering the third threshold, the difference between the user's line of sight direction and the projection direction becomes greater than or equal to the third threshold, so the information display system 1 projects an image 2303 indicating the position or direction of the projection target 1700.

[0114] In this way, the information display system 1 can make information about surrounding objects that user 2 is not looking at more appropriately visible to user 2 by changing one or more of the first threshold, second threshold, and third threshold based on the user 2's line of sight speed.

[0115] [Fourth Embodiment] In the fourth embodiment, an example of the process in which the information display system 1 projects an image onto a projection target based on three-dimensional data, such as data from a digital twin 811 acquired from, for example, an external server 810, will be described.

[0116] Figure 24 is a flowchart showing an example of the projection target determination process according to the fourth embodiment. This process shows, for example, an example of the projection target determination process executed by the projection control unit 803 in step S1404 of Figure 14.

[0117] In step S2401, the projection control unit 803 acquires 3D data including information about the position of the projection target and information to be projected onto the projection target. For example, the projection control unit 803 acquires the 3D data of the digital twin 811 that the information acquisition unit 805 has acquired from the external server 810 and stored in the storage unit 804 from the storage unit 804.

[0118] In step S2402, the projection control unit 803 acquires 3D point cloud data of the area around user 2 from images captured by the surrounding camera or from LiDAR. For example, the projection control unit 803 acquires 3D point cloud data of the area around user 2 by performing VSLAM on the captured images.

[0119] In step S2403, the projection control unit 803 estimates (acquires) the user's position information from the 3D data of the digital twin 811 and the 3D point cloud data of the user's surroundings.

[0120] In step S2404, the projection control unit 803 determines the projection direction based on the projection target and its position obtained from the 3D data of the digital twin 811, and the user's position.

[0121] Through the processing shown in Figure 24, the information display system 1 can project an image 2503 onto the actual projection target 2501 based on the 3D data of the projection target 2501 contained in the digital twin 811, as shown in Figure 25.

[0122] Furthermore, using this method, as shown in Figure 25, even when projecting images onto multiple projection targets 2501 and 2502 with similar shapes, images 2503 and 2504 can be projected more accurately based on information regarding the positions of the multiple projection targets 2501 and 2502.

[0123] As a preferred example, the 3D data of the digital twin 811 includes information indicating the priority of multiple projection targets 2501 and 2502. The projection control unit 803 also modifies the projection method for projecting images 2503 and 2504 onto the multiple projection targets 2501 and 2502 based on the priority of the multiple projection targets 2501 and 2502.

[0124] In the example shown in Figure 25, the information display system 1 projects image 2503 in a blinking display to projection target 2501, which has a higher priority. The information display system 1 also displays image 2504 in a normal display to projection target 2502, which has a lower priority. This allows user 2 to easily understand the importance of images 2503 and 2504 projected onto multiple projection targets 2501 and 2502.

[0125] Furthermore, if multiple projection targets 2501 and 2502 are close together and it is not possible to project images 2503 and 2504 onto both, the projection control unit 803 may project image 2503 only onto the projection target 2501, which has a higher priority.

[0126] As described above, according to each embodiment of the present invention, a wearable terminal worn by the user 2 can be used to appropriately allow the user 2 to see information about surrounding objects that the user 2 is not looking at.

[0127] (supplement) Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0128] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. The present invention is not limited by these embodiments, and the components in these embodiments include those readily conceivable to those skilled in the art, those substantially identical, and those within the scope of so-called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of components can be made without departing from the spirit of these embodiments. [Explanation of Symbols]

[0129] 1. Information display system 2 User 3, 1700, 2501, 2502 Projection targets 100 wearable devices 100-1 The first wearable device 100-2 The second wearable device 300 Information Processing Devices 802 Acquisition Department 803 Projection Control Unit 804 Storage section 807 Projection section [Prior art documents] [Patent Documents]

[0130] [Patent Document 1] Japanese Patent Publication No. 2020-129356

Claims

1. An information display system that allows a user to view information about surrounding objects that the user is not looking at, using a wearable terminal worn by the user, An acquisition unit that acquires detection information including information related to the user's gaze, A projection unit that projects an image onto a projection target in a predetermined projection direction for the user, A projection control unit controls the image projected by the projection unit based on the detection information, It has, The projection control unit, Based on the detection information, the user's effective field of view, stable field of gaze, and peripheral field of view are estimated. If the object is outside the peripheral field of view, the projection unit is controlled to project an image of the object onto a projection target within the effective field of view, the stable field of view, or the peripheral field of view. Information display system.

2. The information display system according to claim 1, wherein the image relating to the object includes a display element indicating the position or direction of the object.

3. The information display system according to claim 1 or 2, wherein the projection control unit projects a predetermined image onto the object with first highlighting when the object is within the stable field of view.

4. The information display system according to claim 3, wherein the projection control unit projects a predetermined image onto the object with a second highlighting that is more emphasized than the first highlighting when the object is outside the stable field of view.

5. The acquisition unit acquires gaze speed information relating to the speed at which the user's gaze direction changes. The projection control unit changes thresholds for estimating the effective field of view, the stable field of gaze, and the peripheral field of view based on the line-of-sight velocity information. The information display system according to any one of claims 1 to 4.

6. The projection control unit, Location information regarding the user's location is acquired, Information regarding the position of the object is obtained from a storage unit that stores information regarding the position of the object and the predetermined image to be projected onto the object. Based on the user's position and the object's position, the projection direction for projecting the predetermined image is determined. The information display system according to claim 3 or 4.

7. The projection unit projects an image onto multiple projection targets, The projection control unit changes the projection method of the image according to the importance of the image to be projected. The information display system according to any one of claims 1 to 6.

8. The information display system according to any one of claims 1 to 7, wherein the acquisition unit acquires the detection information using at least one of an inertial measuring device provided on a wearable terminal worn by the user, or a camera provided on the wearable terminal.

9. The information display system according to any one of claims 1 to 8, wherein the projection unit projects the image onto the projection target using an image projection device provided in a wearable terminal worn by the user.

10. An information display system that uses a wearable terminal worn by the user to make visible to the user information about surrounding objects that the user is not looking at, An acquisition process that acquires detection information including information about the user's gaze, A projection process that projects an image onto a projection target in a predetermined projection direction for the user, Based on the detection information, a projection control process controls the image to be projected in the projection process, Execute, The projection control process is as follows: Based on the detection information, the user's effective field of view, stable field of gaze, and peripheral field of view are estimated. If the object is outside the peripheral field of view, the projection process is controlled to project an image of the object onto a projection target within the effective field of view, the stable field of view, or the peripheral field of view. Information display method.

11. An information display system that uses a wearable terminal worn by the user to make visible to the user information about surrounding objects that the user is not looking at, An acquisition process that acquires detection information including information about the user's gaze, A projection process that projects an image onto a projection target in a predetermined projection direction for the user, Based on the detection information, a projection control process controls the image to be projected in the projection process, Make it run, The projection control process is as follows: Based on the detection information, the user's effective field of view, stable field of gaze, and peripheral field of view are estimated. If the object is outside the peripheral field of view, the projection process is controlled to project an image of the object onto a projection target within the effective field of view, the stable field of view, or the peripheral field of view. program.

Images