Information processing device, information processing system, information processing program, and information processing method

Abstract

When attempting to display virtual objects within the user's field of view, this reduces the possibility of impairing the visibility of the surrounding environment within the user's field of view. [Solution] The information processing device includes: receiving means for receiving data transmission requests from external devices; acquiring means for acquiring the gaze range that the user is fixating on for a predetermined time from the user's eyeball information; notifying means for making a notification regarding the transmission request if the gaze range is wider than a predetermined range, and not making the notification if the gaze range is narrower than a predetermined range; and display control means for controlling the display to display the data for which the transmission request was made after the notification, based on the gaze range, if the gaze range is wider than a predetermined range.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] In recent years, wearable terminals equipped with displays such as head-mounted displays (HMDs) have been increasingly popular. For example, in transmissive AR (Augmented Reality) glasses, information can be overlaid and displayed as virtual objects on the actual scene being viewed through the display in the lens part. When wearing such AR glasses, there is a risk that the virtual objects displayed when the user performs work or the like may block the user's field of view and interfere with the work. To address such issues, there is a technology for controlling the display of virtual objects according to the user's line-of-sight situation. For example, Patent Document 1 discloses such a technology.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, virtual objects are displayed even when a relatively large area of the outdoor scene is viewed from above, which may interfere with the field of view of the user who wants to view the outdoor scene from above.

[0005] The present invention has been made in view of these problems, and an object thereof is to reduce the possibility of impairing the visibility of the outdoor scene within the user's field of view when attempting to display a virtual object within the user's field of view.

Means for Solving the Problems

[0006] To achieve the above objective, the information processing device of the present invention is characterized by comprising: receiving means for receiving a data transmission request from an external device; acquiring means for acquiring a gaze range that the user is fixated on for a predetermined time from the user's eyeball information; notifying means for making a notification regarding the transmission request when the gaze range is wider than a predetermined range, and not making the notification when the gaze range is narrower than a predetermined range; and display control means for controlling the display unit to display the data for which the transmission request was made after the notification, based on the gaze range, when the gaze range is wider than a predetermined range. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technology that can reduce the possibility of impairing the visibility of an image shared by another person when displaying that image. [Brief explanation of the drawing]

[0008] [Figure 1] This is an external view of a wearable terminal 10 according to the first embodiment of the present invention. [Figure 2] This is a block diagram showing an example configuration of a wearable terminal 10 according to the first embodiment of the present invention. [Figure 3] This flowchart uses gaze range information according to the first embodiment of the present invention to determine whether or not it is the right time to receive a shared image from another person, and controls the display accordingly. [Figure 4] This figure illustrates an example of gaze range detection according to the first embodiment of the present invention. [Figure 5] This figure illustrates a notification example and a display example according to the first embodiment of the present invention. [Figure 6] This flowchart, according to the second embodiment of the present invention, determines whether or not it is the right time to receive shared images from another person using gaze range information, and controls the display accordingly. [Figure 7] This figure illustrates an example of displaying another person's image according to a second embodiment of the present invention. [Figure 8] This flowchart, according to the third embodiment of the present invention, determines whether or not it is the right time to receive shared images from others using gaze range information, and controls the display accordingly. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant explanations will be omitted as appropriate. Furthermore, some components, members, and processes will be omitted in each drawing. The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (operating conditions, operating environment, etc.). In the following embodiments, the same components will be denoted by the same reference numerals.

[0010] (First embodiment) <Internal configuration of information processing device> Figure 1 shows an external view of a wearable terminal 10, which is an information processing device to which the present invention can be applied. In particular, AR glasses are given as the wearable terminal in this embodiment. In this embodiment, the information processing device is described as AR glasses, but is not limited to this. It may also be an information processing device such as an HMD, a PC (personal computer) connected to an HMD, a tablet terminal, or a smartphone. Furthermore, it may be an information processing system equipped with each of the components of the information processing device of this embodiment.

[0011] In Figure 1, the right-eye display unit 160R and the left-eye display unit 160L each include an eyepiece 16, a display 29, and an eyeball detection unit 161, as described later. The user can view the image displayed on the internal display 29 through the eyepiece 16. The display 29 is a transmissive display unit and is positioned to cover the user's field of view. The user perceives the image (virtual image) displayed on the display 29 as being superimposed on their real field of view.

[0012] The right-eye imaging unit 150R and the left-eye imaging unit 150L each include a lens 103 and an imaging unit 22, as described later, and are arranged to capture an area corresponding to the user's field of view. The captured images for the right and left eyes can also be displayed on the right-eye display unit 160R and the left-eye display unit 160L, respectively. The imaging unit 150R and the imaging unit 150L each have a zoom mechanism, and the zoom magnification can be changed using the operation unit 70. The zoom operation can be a combination of optical zoom using the lens zoom mechanism and electronic zoom by cropping the captured image.

[0013] The power switch 72 is an operating component that switches the power of the wearable terminal 10 ON and OFF.

[0014] Speaker 90 is a speaker that provides various voice notifications to the user when the device is being worn.

[0015] Figure 2 is a block diagram showing an example configuration of the wearable terminal 10 according to this embodiment.

[0016] The system control unit 50 is a control unit consisting of at least one processor or circuit, and controls the entire wearable terminal 10. It implements each of the processes of this embodiment, which will be described later, by executing a program recorded in the non-volatile memory 56, which will be described later. For example, RAM is used in the system memory 52, and constants, variables for the operation of the system control unit 50, programs read from the non-volatile memory 56, etc. are stored there. The system control unit 50 also performs display control by controlling the memory 32, the display 29, etc. For example, it changes the display on the display 29 based on the determination result of the user state determination unit 165, which will be described later. The system control unit 50 also notifies the user by display or sound that there has been a request to share (transmit) an image from another person, based on the determination result of the user state determination unit 165, which will be described later. Note that the means of notification are not limited to display or sound. Alternatively, instead of the system control unit 50 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0017] The non-volatile memory 56 is an electrically erasable and recordable memory, and for example, a Flash-ROM or the like is used. In the non-volatile memory 56, constants for the operation of the system control unit 50, programs, and the like are stored. Here, the program refers to a program for executing various flowcharts described later in this embodiment.

[0018] The lens 103 is composed of a plurality of lenses, but in FIG. 2, it is shown simply as a single lens for simplicity. The system control unit 50 communicates with the lens system control circuit 4 and controls the aperture 1 via the aperture drive circuit 2 to achieve aperture control. Also, the focus lens in the lens 103 is displaced via the AF drive circuit 3 to focus on the subject.

[0019] The imaging unit 22 is an image sensor composed of a CCD, a CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 incorporates an A / D converter and is used to convert the analog signal output from the imaging unit 22 into a digital signal. Imaging by the imaging unit 22 is synchronized with the horizontal synchronization signal and the vertical line synchronization signal output from a timing generator unit (not shown), and image data for one frame is output as frame data at the period of the vertical line synchronization signal. The imaging unit 22 is a frame-based synchronous sensor with respect to the event sensor 163, which is an asynchronous event-based sensor described later.

[0020] The image processing unit 24 performs resizing and color conversion processing, such as predetermined pixel interpolation and reduction, on the data from the imaging unit 22 or the data from the memory control unit 15 (described later). The image processing unit 24 also performs predetermined calculations using the captured image data. Based on the calculation results obtained by the image processing unit 24, the system control unit 50 performs exposure control and distance measurement control. This enables TTL (through-the-lens) AF (autofocus) processing and AE (automatic exposure) processing. The image processing unit 24 further performs predetermined calculations using the captured image data and performs TTL AWB (auto white balance) processing based on the calculation results obtained. In addition, picture style processing is possible, which allows selective setting of shooting parameters such as whether the captured image is a color image or a monochrome image.

[0021] The memory control unit 15 controls the transmission and reception of data between the imaging unit 22, the image processing unit 24, and the memory 32. Output data from the imaging unit 22 is written to the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and image data for display on the display 29.

[0022] Furthermore, memory 32 also serves as memory for image display (video memory). Image data for display written to memory 32 is displayed by the display 29 via the memory control unit 15.

[0023] The display 29 displays information on an LCD or organic EL display device according to the signal from the memory control unit 15. By sequentially transferring the image data stored in the memory 32 to the display 29 and displaying it, a continuous display of the captured image can be achieved.

[0024] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of the built-in clock.

[0025] The operation unit 70 consists of various operating components that act as input units for receiving user input. The operation unit 70 includes, for example, a voice UI (User Interface) and a touchpad. The touchpad is mounted on the side (not shown) of the wearable terminal 10. The system control unit 50 can detect operations or states on the touchpad. The position coordinates of the finger touching the touchpad are notified to the system control unit 50 via the internal bus, and the system control unit 50 determines what kind of operation (touch operation) was performed on the touchpad based on the notified information. The touchpad may use any type of touch panel from various types such as resistive, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, and optical sensor types, and two or more different types of touch panels may be calibrated.

[0026] The power control unit 31 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, and the like, and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 31 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage to each part, including the recording medium 200, for the required period of time. The power supply unit 30 consists of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and lithium-ion batteries, an AC adapter, and the like.

[0027] The recording medium I / F17 is an interface to the recording medium 200, such as a memory card or hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of semiconductor memory, magnetic disks, etc.

[0028] The communication unit 54 connects wirelessly or via a wired cable to transmit and receive data such as video and audio signals with external devices. The communication unit 54 can also connect to a wireless LAN (Local Area Network) or the internet. Furthermore, the communication unit 54 can communicate with external devices using Wi-Fi (Wireless Fidelity) (registered trademark), Bluetooth (registered trademark), and Bluetooth Low Energy.

[0029] The eyepiece detection unit 57 is a wear detection sensor that detects whether or not a user is wearing the wearable terminal 10. The system control unit 50 can switch the wearable terminal 10 on (power on) / off (power off) according to the state detected by the eyepiece detection unit 57. The eyepiece detection unit 57 can use, for example, an infrared proximity sensor, and may be configured to detect the approach of some object to the eyepiece 16. When an object approaches, infrared light emitted from the light-emitting part (not shown) of the eyepiece detection unit 57 is reflected and received by the light-receiving part (not shown) of the infrared proximity sensor. The amount of infrared light received can be used to determine how close the object is to the eyepiece 16. Note that the infrared proximity sensor is just one example, and other sensors such as a capacitive type may be used.

[0030] The eyeball detection unit 161 consists of an eyeball detection lens 162 (described later), an event sensor 163, and an event data calculation unit 164, and is capable of detecting eyeball information of a user wearing the wearable terminal 10.

[0031] The infrared light emitted from the infrared light-emitting diode 58 is reflected by the eyeball (eye) 166, and this reflected infrared light is imaged onto the imaging surface of the event sensor 163 through the eyeball detection lens 162.

[0032] The event sensor 163 is an event-based vision sensor that detects changes in the brightness of light incident on each pixel and outputs information about the pixel where the brightness change occurred, asynchronously from other pixels. The data output from the event sensor 163 (hereinafter referred to as event data) includes, for example, the position coordinates of the pixel where the brightness change (event) occurred, the polarity (positive or negative) of the brightness change, and timing information corresponding to the time the event occurred. Compared to existing frame-based synchronous sensors such as the imaging unit 22, the event sensor 163 eliminates redundancy in the output information and features high-speed operation, high dynamic range, and low power consumption. On the other hand, the event data is asynchronously output for each pixel. In order to determine the relationship between event data, it is necessary to accumulate event data that occurs over a predetermined period of time and perform various calculations on the results.

[0033] The event data calculation unit 164 is a calculation unit for detecting eyeball information based on event data output continuously and asynchronously from the event sensor 163. For example, it accumulates event data that occurs over a predetermined period of time and processes them as a set of data to determine the presence or absence of eyeball information. By changing the accumulation time for accumulating event data, it is possible to detect multiple eyeball information items with different occurrence speeds. Eyeball information includes eye movements, such as gaze position information, gaze trajectory, saccade information including the direction and speed of saccades, and microsaccade information including the frequency and amplitude of microsaccades. It may also include pupil information including pupil size and its changes, and blink information such as blink speed and number of blinks. The detectable eyeball information is merely an example and is not limited to these. The event data calculation unit 164 may be configured to map the event data for the accumulation time as image data for one frame based on the event occurrence coordinates and perform image processing. This makes it possible to obtain multiple eyeball information items from event data using the method conventionally used in frame-based image data processing.

[0034] The user state determination unit 165 is a determination unit that determines the user's state based on eyeball information detected by the event data calculation unit 164. For example, it can determine the gaze range or degree of gaze (overall view) from the frequency and amplitude of microsaccades. Here, gaze range is synonymous with attention range and focus range. The degree of gaze is an index that is higher the narrower the gaze range and lower the wider it is. Overall view is defined as the opposite of degree of gaze. In addition, it can determine the user's level of concentration (state of concentration) or fatigue from the frequency and amplitude of microsaccades, pupil size and change amount, and blinking speed and number. Furthermore, preferences such as facial preference are related to the speed of microsaccades and pupil diameter, and can be determined from both parameters. The user state determination unit 165 can be composed of a neural network or the like that takes parameters related to microsaccades, blinking, pupil, and the identification results of the subject identification unit 70 as inputs and outputs an index representing the user state (hereinafter referred to as user state information). User state information includes, for example, gaze range, degree of gaze (antonym: degree of overview), degree of concentration (antonym: degree of fatigue), preference level, etc. However, the configuration of the user state determination unit 165 is not limited to the above, and other determination means may be used. The eyeball information and determination results used in the user state determination unit 165 are not limited to the above, and other configurations may be used.

[0035] The eye-tracking input setting unit 167, via the system control unit 50, sets whether eye-tracking detection by the eyeball detection unit 161 is enabled or disabled. Alternatively, it can set parameters and detection conditions related to the event data calculation unit 164 and the user state determination unit 165. These can be arbitrarily set by the user via menu settings.

[0036] Furthermore, the system control unit 50 can acquire information on which area of ​​the display 29 the captured subject (real object) is displayed in and at what size. In addition, the system control unit 50 can also acquire information on which area of ​​the display 29 the user is looking at, using the eyeball detection unit 161. This allows the system control unit 50 to determine which area of ​​the subject the user is looking at.

[0037] <A process that determines whether or not it is time to receive image sharing from others and controls the display accordingly.> The following describes an example of a first embodiment of the present invention in which the timing for receiving an image from another person is determined using gaze range information, and the display is controlled, with reference to Figures 3, 4, and 5.

[0038] Referring to Figure 3, a flowchart will be described that determines whether or not it is time to receive shared images from others using gaze range information according to the first embodiment of the present invention, and controls the display. Each process in the flowchart of Figure 3 is realized by the system control unit 50 loading a program stored in the non-volatile memory 56 into the system memory 52 and executing it, thereby controlling each functional block. The flowchart of Figure 3 starts when the wearable terminal 10 is started up and the display image data written to the memory 32 becomes displayable on the display 29 via the memory control unit 15.

[0039] In step S301, the system control unit 50 determines, via the communication unit 54, whether or not there is a request to share an image from an external device. If the system control unit 50 determines that there is a request to share an image from an external device, it proceeds to step S302; if it determines that there is no request to share an image from an external device, it proceeds to step S301. In other words, if it determines that there is no request to share an image from an external device, the process in step S301 is repeated. Note that, in this explanation, we have used the example of sharing an image from another person, and therefore referred to it as a request to share an image, but it is a request to send an image from another person, and can also be expressed as a request to send an image.

[0040] In step S302, the system control unit 50 refers to the user state determination unit 165 to determine whether or not it has detected gaze within a gaze range smaller than a predetermined size. If the system control unit 50 determines that it has detected gaze within a gaze range smaller than a predetermined size, it proceeds to step S303; otherwise, it proceeds to step S306. The method for detecting the gaze range will be described later.

[0041] In step S303, the system control unit 50 receives an image from an external device via the communication unit 54, displays the received image on the display 29, and proceeds to step S304. The system control unit 50 displays the received image so that it does not overlap with the user's viewing area. An example of the display will be described later.

[0042] In step S304, the system control unit 50 determines whether to terminate the display of the image received from the external device via the communication unit 54. If the system control unit 50 determines that the reception of the image has ended, it proceeds to step S305; otherwise, it proceeds to step S304. Here, the system control unit 50 determines, for example, whether the user has given an instruction to terminate the display of the image received from the external device via the operation unit 70. Alternatively, it may determine whether the image received from the external device has been displayed for a predetermined period of time.

[0043] In step S305, the system control unit 50 terminates the display of the image received from the external device via the communication unit 54 and ends this process.

[0044] In step S306, the system control unit 50 displays a notification to the user that an image sharing request has been received from an external device, and then proceeds to step S307. An example of the notification will be described later.

[0045] In step S307, the system control unit 50 determines whether the user has permitted the display of the image within a predetermined time after the notification display started in step S306. If the system control unit 50 determines that the user has permitted the display of the image, it proceeds to step S303; otherwise, it proceeds to step S308. The user selects whether to permit or deny the display of the image, for example, via the operation unit 70. That is, if permission to display the image is selected within a predetermined time after the notification display starts, the system control unit 50 determines that the user has permitted the display of the image. In other words, it obtains an instruction from the user to display the image (instruction acquisition step). Also, if denial of image display is selected within a predetermined time after the notification display starts, or if neither permission nor denial of image display is selected within a predetermined time after the notification display starts, the system control unit 50 does not determine that the user has permitted the display of the image.

[0046] In step S308, the system control unit 50 does not display the image received from the external device via the communication unit 54, and proceeds to step S309.

[0047] In step S309, the system control unit 50 notifies the external device via the communication unit 54 that it is not possible to display the received image, and terminates this process.

[0048] In the process shown in Figure 3, before receiving an image from an external device, it is determined whether or not to allow its display. If display is not allowed, the image is not received from the external device. However, even if display of an image received from an external device is not allowed, the image may still be received but not displayed. This allows the user to view the image received from the external device at a desired time later, even if they cannot view it in real time.

[0049] <Specific examples of gaze range detection> Next, an example of gaze range detection according to the first embodiment of the present invention will be described with reference to Figures 4(a), 4(b), 4(c), and 4(d). Here, Figures 4(a) and 4(b) are microsaccade information, which is one of the eyeball information acquired by the event data calculation unit 164. Figures 4(c) and 4(d) are diagrams representing the user's gaze state on the display 29.

[0050] Figure 4(a) shows the microsaccade waveform when the gaze range is relatively wide. The vertical axis represents the position of the pupil center in the ocular-center coordinate system (expressed as an angle), and the horizontal axis represents time. 401 shows the change in eye position when a microsaccade occurs. The wider the gaze range, the larger the amplitude of eye movement and the higher the oscillation (lower the attenuation rate) tend to be. Also, the frequency of microsaccade occurrences over a certain period, as shown in 402, tends to be higher.

[0051] On the other hand, Figure 4(b) shows the microsaccade waveform when the gaze range is relatively narrow. 403 shows the change in eye position per unit time when microsaccades occur. The narrower the gaze range, the smaller the amplitude of eye movement and the lower the oscillation (higher the attenuation rate) tend to be. Also, the frequency of microsaccade occurrence tends to be lower in 404, which represents the same period as 402. By using these eye movement characteristics as parameters, the width of the gaze range can be estimated.

[0052] Figure 4(c) shows the user's gaze state when the gaze range is relatively wide, as in Figure 4(a). Image 410 shows the area visible to the user via the wearable terminal 10, and includes the first object 406 and the second object 407. The first gaze range 405 is shown by a dotted line, which represents the gaze range of the predetermined size mentioned above. The user's gaze range 408 is shown by a solid line, and the user's gaze range 408 is larger than the first gaze range 405. This assumes a state where the user is looking over relatively far ahead. In this case, in the process of Figure 3 described above, in step S302, the system control unit 50 determines that it has detected gaze in a gaze range larger than the predetermined size. That is, the user is not gazing (looking over) as much as the standard degree of gaze.

[0053] Next, Figure 4(d) shows the user's gaze state when the gaze range is relatively narrow, as in Figure 4(b). The user's gaze range 409 is shown by a solid line, and the user's gaze range 409 is smaller than the first gaze range 405. This assumes a state in which the user is relatively focused, and that the user is focused on the first object 406. In this case, in the process of Figure 3 described above, in step S302, the system control unit 50 determines that it has detected gaze within a gaze range smaller than a predetermined size. That is, the user is focusing more than the standard degree of focus.

[0054] <Specific examples of notifications and displays> Figures 5(a), 5(b), and 5(c) show examples of notifications and displays according to embodiments of the present invention. Here, the positions of the gaze range 501a, gaze range 501b, image 503a, and image 503b are calculated as a two-dimensional coordinate system on the display 29. For example, the starting point is the lower left and the coordinate resolution is (x,y)=(1280,720). Note that a coordinate system other than two dimensions may also be used.

[0055] Figure 5(a) shows an example of displaying a notification when gaze is detected in a gaze area larger than a predetermined size. The user's gaze area 501a on the display 29 is larger than the first gaze area 405. At this time, it is determined that the user is looking around, and if there is a request to share an image from an external device, a notification display 502 is displayed to notify the user. The notification may be an icon display like the notification display 502, or a banner display. In addition, notifications may be made by methods other than display, such as sound or vibration.

[0056] Figure 5(b) shows an example of display when the user receives a notification 502 and authorizes the display of an image received from an external device. In Figure 5(b), the gaze area 501a is included in the region enclosed by the bottom left point (x1,y1)=(400,100), the bottom right point (x2,y2)=(800,100), the top left point (x3,y3)=(400,600), and the top right point (x4,y4)=(800,600). The image 503a received from the external device is displayed so as not to overlap with the gaze area 501a. In Figure 5(b), image 503a is displayed in the region with the four corners (x1,y1)=(50,200), (x2,y2)=(350,200), (x3,y3)=(50,500), and (x4,y4)=(350,500). By positioning image 503a and the gaze area 501a so as not to overlap, the user's field of view can be ensured.

[0057] Figure 5(c) shows an example of displaying an image transmitted from an external device when a gaze is detected within a gaze area smaller than a predetermined size. In Figure 5(c), the gaze area 501b is included in the region enclosed by the bottom left point (x1,y1)=(600,200), the bottom right point (x2,y2)=(700,200), the top left point (x3,y3)=(600,300), and the top right point (x4,y4)=(700,300). The image 503b received from the external device is displayed so as not to overlap with the gaze area 501b. In Figure 5(c), image 503b is displayed in the region with the four corners (x1,y1)=(50,100), (x2,y2)=(500,100), (x3,y3)=(50,600), and (x4,y4)=(500,600). By positioning image 503b and the gaze area 501b so that they do not overlap, the user's field of view can be ensured.

[0058] The size at which the shared image is displayed may be determined based on the gaze area. Alternatively, it may be displayed at the largest size possible without overlapping the gaze area. Furthermore, the size at which the shared image is displayed may be determined by the external device.

[0059] As described above, according to the first embodiment, the system uses gaze range information to determine whether or not it is time to receive a shared image from another person, and controls the display accordingly. This allows the user to receive a shared image from another person without impairing the visibility of the user's field of view.

[0060] (Second Embodiment) Hereinafter, with reference to Figures 6 and 7, an example of a second embodiment of the present invention will be described in which the timing for receiving a shared image from another person is determined using gaze range information, and the display is controlled accordingly.

[0061] <A workflow that uses gaze range information to determine whether or not it is appropriate to receive shared images from others and controls the display accordingly.> Figure 6 is a flowchart that determines whether or not it is time to receive shared images from others using gaze range information according to the second embodiment of the present invention, and controls the display. Each process in the flowchart of Figure 6 is realized by the system control unit 50 loading the program stored in the non-volatile memory 56 into the system memory 52 and executing it, and controlling each functional block. Steps identical to those in the flowchart of Figure 3 are indicated by the same symbols, and steps S301 to S302, S305, and S306 to S309 are the same processes, so their explanation is omitted. The flowchart of Figure 6 starts when the wearable terminal 10 is started up and the display image data written to the memory 32 becomes displayable on the display 29 via the memory control unit 15.

[0062] In step S303, the system control unit 50 receives an image from an external device via the communication unit 54, displays the received image on the display 29, and proceeds to step S601. The system control unit 50 displays the received image so that it does not overlap with the user's viewing area.

[0063] In step S601, the system control unit 50 determines whether the area displaying the image received from the external device overlaps with the newly estimated gaze area. If the system control unit 50 determines that the area displaying the image received from the external device overlaps with the newly estimated gaze area, it proceeds to step S602. If the system control unit 50 does not determine that the area displaying the image received from the external device overlaps with the newly estimated gaze area, it proceeds to step S304.

[0064] In step S602, the system control unit 50 determines whether or not the user is attempting to view an image received from an external device. If the system control unit 50 determines that the user is attempting to view an image received from an external device, it proceeds to step S304; otherwise, it proceeds to step S603. Whether or not the user is attempting to view an image received from an external device can be determined, for example, by calculating information about the depth of the line of sight and determining whether the depth of the user's viewing position is within the depth of the position where the image received from the external device is displayed. Information about the depth of the line of sight can be calculated, for example, using the convergence angle, using a known method. Alternatively, whether or not the user is attempting to view an image received from an external device can also be determined based on the gaze range. For example, even if the gaze range changes, eye movements may unintentionally cause a portion of the gaze range to overlap with the area displaying the image received from the external device. Therefore, if the newly estimated gaze range and the area displaying the image received from the external device overlap over a wider area than a predetermined area, it may be determined that the user is attempting to view an image received from an external device.

[0065] In step S603, the system control unit 50 changes the position and size of the image received from the external device so that it does not overlap with the newly estimated gaze range, and then proceeds to step S304. Here, the priority given to changing the position or changing the size may be predetermined. An example of the display will be described later.

[0066] In step S304, the system control unit 50 determines whether or not to terminate the display of the image received from the external device via the communication unit 54. If the system control unit 50 determines that the reception of the image has ended, it proceeds to step S305; otherwise, it proceeds to step S601. Here, the system control unit 50 determines, for example, whether or not the user has given an instruction to terminate the display of the image received from the external device via the operation unit 70.

[0067] <Specific examples of display> Figures 7(a) and 7(b) show examples of displays according to the second embodiment of the present invention.

[0068] Figure 7(a) assumes a scenario where the gaze range is changed from gaze range 501a at the time of Figure 5(b) to gaze range 701a. Originally, the image received from the external device was displayed at the position and size of image 503a, but when the gaze range is changed to gaze range 701a, an overlapping area is created between gaze range 701a and image 503a. In Figure 7(a), in such a case, the size of the image received from the external device is changed to display as shown in image 702a so that it does not overlap with gaze range 701a. Specifically, the area of ​​gaze range 701a is included in the area enclosed by the bottom left point (x1,y1)=(300,100), bottom right point (x2,y2)=(700,100), top left point (x3,y3)=(300,600), and top right point (x4,y4)=(700,600). Therefore, image 702a is modified to be displayed in the area with the bottom left corner (x1,y1)=(50,400), the bottom right corner (x2,y2)=(200,400), the top left corner (x3,y3)=(50,600), and the top right corner (x4,y4)=(200,600). In this way, by changing the position of the image in accordance with changes in the user's gaze range, the user's field of view can be ensured.

[0069] Figure 7(b) assumes a scenario where the gaze range is changed from gaze range 501b at the time of Figure 5(c) to gaze range 701b. Originally, the image received from the external device was displayed at the position and size of image 503b, but when the gaze range is changed to gaze range 701b, an overlapping area is created between gaze range 701b and image 503b. In Figure 7(b), in such a case, the size of the image received from the external device is changed to that of image 702b so that it does not overlap with gaze range 701b. Specifically, the area of ​​gaze range 701b is included in the area enclosed by the bottom left point (x1,y1)=(400,180), bottom right point (x2,y2)=(550,180), top left point (x3,y3)=(400,360), and top right point (x4,y4)=(550,360). Therefore, image 702b is modified to be displayed in an area with the four corners at (x1,y1)=(780,100), (x2,y2)=(1230,100), (x3,y3)=(780,600), and (x4,y4)=(1230,600). In this way, by changing the position and size of the image in accordance with changes in the user's gaze range, the user's field of view can be ensured.

[0070] (Third embodiment) The following describes an example of determining whether or not it is time to receive a shared image from another person using gaze range information according to the third embodiment of the present invention, and controlling the display accordingly, with reference to Figure 8.

[0071] <A workflow that uses gaze range information to determine whether or not it is appropriate to receive shared images from others and controls the display accordingly.> Figure 8 is a flowchart that determines whether or not it is time to receive shared images from others using gaze range information according to the third embodiment of the present invention, and controls the display. Each process in the flowchart of Figure 8 is realized by the system control unit 50 loading the program stored in the non-volatile memory 56 into the system memory 52 and executing it, thereby controlling each functional block. Steps identical to those in the flowchart of Figure 3 are indicated by the same symbols, and steps S301 to S302, S305, and S306 to S309 are identical processes, so their explanation is omitted. The flowchart of Figure 8 starts when the wearable terminal 10 is started up and the display image data written to the memory 32 becomes displayable on the display 29 via the memory control unit 15.

[0072] In step S303, the system control unit 50 receives an image from an external device via the communication unit 54, displays the received image on the display 29, and proceeds to step S801. The system control unit 50 displays the received image so as not to overlap with the user's viewing area.

[0073] In step S801, the system control unit 50 determines whether or not there is an image sharing request from a second external device different from the external device that made the sharing request in step S301. If the system control unit 50 determines that there is an image sharing request from the second external device, it proceeds to step S802; otherwise, it proceeds to step S304.

[0074] In step S802, the system control unit 50 does not display the image received from the second external device and proceeds to step S803. Here, the system control unit 50 may choose not to receive the image from the second external device. Alternatively, the system control unit 50 may receive the image from the second external device, notify the user, and allow the user to review it later.

[0075] In step S803, the system control unit 50 notifies the second external device via the communication unit 54 that it is not possible to display the image, and proceeds to step S304.

[0076] In step S304, the system control unit 50 determines whether or not to terminate the display of the image received from the external device via the communication unit 54. If the system control unit 50 determines that the reception of the image has ended, it proceeds to step S305; otherwise, it proceeds to step S801. Here, the system control unit 50 determines, for example, whether or not the user has given an instruction to terminate the display of the image received from the external device via the operation unit 70.

[0077] As described above, according to the third embodiment, the gaze range information is used to determine whether or not it is the right time to receive shared images of others among multiple devices, and the display is controlled accordingly. This allows users to receive shared images of others without impairing the visibility of their field of view.

[0078] (Variation 1) In the first, second, and third embodiments described above, examples were explained in which images received from an external device are displayed so as not to overlap with the gaze range. In this case, even if the user becomes interested in the image received from the external device and tries to gaze at it, the display may be changed so that the image received from the external device is out of the gaze range, and the user may not be able to properly view the image received from the external device. Therefore, while the image received from the external device is being displayed, the user may be able to give an instruction via the operation unit 70 to fix the display of the image received from the external device. In addition, the user may be able to change the position and size of the image received from the external device via the operation unit 70.

[0079] (Modification 2) In the first, second, and third embodiments described above, examples were explained in which the external device is notified that it is unable to display the image that was requested to be shared from the external device, but the invention is not limited to these examples. For example, considering a Social Networking Service (SNS), there are cases where images are shared in one-on-one communication, to a group of people, or to an unspecified number of people. In the case of one-on-one communication or to a group of people, it is likely that the people involved are close, and it may be convenient to notify them that the image cannot be displayed. However, in the case of sharing an image to an unspecified number of people, if the external device that shared the image is notified by an unspecified number of devices that it cannot be displayed, the amount of notifications on the external device may increase. Therefore, in the case of sharing an image to an unspecified number of people, it may be acceptable not to notify the external device that it cannot be displayed.

[0080] (Other embodiments) Furthermore, the present invention can also be realized by performing the following process: that is, supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a network or various storage media, and having the computer (or control unit or MPU, etc.) of the system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0081] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Some of the above embodiments may be combined as appropriate.

[0082] Furthermore, each functional unit in each of the above embodiments (each modified example) may or may not be individual hardware. The functions of two or more functional units may be implemented by common hardware. Each of the multiple functions of a single functional unit may be implemented by individual hardware. Two or more functions of a single functional unit may be implemented by common hardware. In addition, each functional unit may or may not be implemented by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. The functions of at least some of the functional units of the device may be implemented by the processor reading and executing the control program from the memory.

[0083] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0084] Furthermore, in each of the examples described above, "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPUs) and specialized processors (e.g., GPUs, ASICs, FPGAs, and programmable logic devices, etc.).

[0085] This embodiment includes the following configurations, methods, and programs.

[0086] [Configuration 1] A receiving means for receiving data transmission requests from external devices, An acquisition means for acquiring the gaze range that the user is fixating on for a predetermined period of time from the user's eyeball information, A notification means that, if the gaze range is wider than a predetermined range, provides notification regarding the transmission request, and if the gaze range is narrower than the predetermined range, does not provide notification. If the gaze range is wider than the predetermined range, the display control means controls the display unit to display the data for which transmission was requested after the notification, based on the gaze range. An information processing device characterized by the following:

[0087] [Configuration 2] The acquisition means acquires the gaze range for the predetermined time period from the time the transmission request is received by the receiving means. The information processing device according to configuration 1, characterized by the above.

[0088] [Configuration 3] The notification means determines whether or not to make the notification based on the gaze range acquired by the acquisition means by the time the transmission request is received by the receiving means. An information processing apparatus according to configuration 1 or 2, characterized by the above.

[0089] [Structure 4] The notification means provides the notification by displaying an icon on the display unit or by sound. An information processing device according to any one of configurations 1 to 3.

[0090] [Composition 5] The aforementioned data is an image, The display control means displays the image so as not to overlap with the gaze area. An information processing apparatus according to any one of configurations 1 to 4, characterized by the above.

[0091] [Composition 6] The aforementioned data is an image, The display control means determines the size of the image to be displayed based on the gaze range. An information processing device according to any one of configurations 1 to 5, characterized by the above.

[0092] [Composition 7] The system further includes an instruction acquisition means for acquiring an instruction from the user to display the data that has been requested to be transmitted, When the instruction is acquired by the instruction acquisition means, the display control means controls the display unit to display the data for which transmission was requested. An information processing device according to any one of configurations 1 to 6.

[0093] [Structure 8] If the instruction is not acquired by the instruction acquisition means, the display control means controls the data for which transmission was requested not to be displayed on the display unit. The information processing apparatus according to configuration 7, characterized by the features described above.

[0094] [Composition 9] If the data for which transmission was requested is not displayed on the display unit, the system further includes a transmission means for transmitting a second notification to the external device indicating that the data cannot be displayed on the display unit. An information processing device according to any one of configurations 1 to 8.

[0095] [Configuration 10] If, while the display unit is displaying the first data requested for transmission from the first external device, the receiving means receives a request for transmission of second data (image) from a second external device different from the first external device, the transmitting means transmits the second notification to the second external device. The information processing apparatus according to configuration 9, characterized by the features described therein.

[0096] [Composition 11] The transmission means transmits the second notification to the external device if the transmission request from the external device is a transmission request directed only to the information processing device, and the requested data is not displayed on the display unit. However, if the transmission request from the external device is not a transmission request directed only to the information processing device, the transmission means does not transmit the second notification to the external device, even if the requested data is not displayed on the display unit. The information processing apparatus according to configuration 9 or 10, characterized by the above.

[0097] [Composition 12] The transmission means shall not transmit the second notification to the external device if the transmission request from the external device is a transmission request to an unspecified number of people, including the information processing device, and the data for which the transmission request was made is not displayed on the display unit. An information processing apparatus according to any one of the configurations 9 to 11, characterized by the above.

[0098] [Composition 13] The display control means controls the display unit to display the requested data without issuing a notification if the gaze range is narrower than the predetermined range. An information processing device according to any one of configurations 1 to 12, characterized by the above.

[0099] [Composition 14] The eyeball information includes at least one of the following: the frequency of microsaccade occurrence, the amplitude of the microsaccade, the position of the line of sight, the direction of the saccade, the speed of the saccade, the size of the pupil, and the change in pupil diameter. An information processing device according to any one of configurations 1 to 13, characterized by the above.

[0100] [Composition 15] The gaze range is the range that includes the trajectory of the user's gaze during the predetermined time. An information processing apparatus according to any one of configurations 1 to 14, characterized by the features described herein.

[0101] [method] A receiving step in which a data transmission request is received from an external device, An acquisition step of obtaining the gaze range that the user is fixating on for a predetermined time from the user's eyeball information, A notification step in which, if the gaze range is wider than a predetermined range, a notification regarding the transmission request is made, and if the gaze range is narrower than the predetermined range, the notification is not made. If the gaze range is wider than the predetermined range, the display control step includes controlling the display unit to display the data for which transmission was requested after the notification, based on the gaze range. An information processing method characterized by the following:

[0102] [program] A program for causing a computer to function as one of the means of an information processing device described in any one of items 1 to 15.

[0103] [system] External devices and Display device and A receiving device that receives a data transmission request from the aforementioned external device, An acquisition device that acquires the gaze range that the user fixates on for a predetermined period of time from the user's eyeball information, A notification device that, if the gaze range is wider than a predetermined range, makes a notification regarding the transmission request, and if the gaze range is narrower than the predetermined range, does not make the notification. The display control device, if the gaze range is wider than the predetermined range, controls the display to display the data for which transmission was requested on the display device after the notification, based on the gaze range. An information processing system characterized by the following:

Claims

1. A receiving means for receiving data transmission requests from external devices, An acquisition means for acquiring the gaze range that the user is fixating on for a predetermined period of time from the user's eyeball information, A notification means that, if the gaze range is wider than a predetermined range, provides notification regarding the transmission request, and if the gaze range is narrower than the predetermined range, does not provide notification. If the gaze range is wider than the predetermined range, the display control means controls the display unit to display the data for which transmission was requested after the notification, based on the gaze range. An information processing device characterized by the following:

2. The acquisition means acquires the gaze range for the predetermined time period from the time the transmission request is received by the receiving means. The information processing apparatus according to feature 1.

3. The notification means determines whether or not to make the notification based on the gaze range acquired by the acquisition means by the time the transmission request is received by the receiving means. The information processing apparatus according to feature 1.

4. The notification means provides the notification by displaying an icon on the display unit or by sound. The information processing apparatus according to feature 1.

5. The aforementioned data is an image, The display control means displays the image so as not to overlap with the gaze area. The information processing apparatus according to feature 1.

6. The aforementioned data is an image, The display control means determines the size of the image to be displayed based on the gaze range. The information processing apparatus according to feature 1.

7. The system further includes an instruction acquisition means for acquiring an instruction from the user to display the data that has been requested to be transmitted, When the instruction is acquired by the instruction acquisition means, the display control means controls the display unit to display the data for which transmission was requested. The information processing apparatus according to feature 1.

8. If the instruction is not acquired by the instruction acquisition means, the display control means controls the data for which transmission was requested not to be displayed on the display unit. The information processing apparatus according to feature 7.

9. If the data for which transmission was requested is not displayed on the display unit, the system further includes a transmission means for transmitting a second notification to the external device indicating that the data cannot be displayed on the display unit. The information processing apparatus according to feature 1.

10. If, while the display unit is displaying the first data requested for transmission from the first external device, the receiving means receives a request for transmission of second data (image) from a second external device different from the first external device, the transmitting means transmits the second notification to the second external device. The information processing apparatus according to feature 9.

11. The transmission means transmits the second notification to the external device if the transmission request from the external device is a transmission request directed only to the information processing device, and the requested data is not displayed on the display unit. However, if the transmission request from the external device is not a transmission request directed only to the information processing device, the transmission means does not transmit the second notification to the external device, even if the requested data is not displayed on the display unit. The information processing apparatus according to feature 9.

12. The transmission means shall not transmit the second notification to the external device if the transmission request from the external device is a transmission request to an unspecified number of people, including the information processing device, and the data for which the transmission request was made is not displayed on the display unit. The information processing apparatus according to feature 9.

13. The display control means controls the display unit to display the requested data without issuing a notification if the gaze range is narrower than the predetermined range. The information processing apparatus according to feature 1.

14. The eyeball information includes at least one of the following: the frequency of microsaccade occurrence, the amplitude of the microsaccade, the position of the line of sight, the direction of the saccade, the speed of the saccade, the size of the pupil, and the change in pupil diameter. The information processing apparatus according to feature 1.

15. The gaze range is the range that includes the trajectory of the user's gaze during the predetermined time. The information processing apparatus according to feature 1.

16. A receiving step in which a data transmission request is received from an external device, An acquisition step of obtaining the gaze range that the user is fixating on for a predetermined time from the user's eyeball information, A notification step in which, if the gaze range is wider than a predetermined range, a notification regarding the transmission request is made, and if the gaze range is narrower than the predetermined range, the notification is not made. If the gaze range is wider than the predetermined range, the display control step includes controlling the display unit to display the data for which transmission was requested after the notification, based on the gaze range. An information processing method characterized by the following:

17. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

18. External devices and Display device and A receiving device that receives a data transmission request from the aforementioned external device, An acquisition device that acquires the gaze range that the user fixates on for a predetermined period of time from the user's eyeball information, A notification device that, if the gaze range is wider than a predetermined range, makes a notification regarding the transmission request, and if the gaze range is narrower than the predetermined range, does not make the notification. The display control device, if the gaze range is wider than the predetermined range, controls the display to display the data for which transmission was requested on the display device after the notification, based on the gaze range. An information processing system characterized by the following:

Images