Electronic device, method of controlling the same, and program

Abstract

To provide a technique that enables efficient reduction in processing load on a device without sacrificing user convenience.SOLUTION: An electronic device of the present invention comprises acquisition means for acquiring information related to the condition of eyes of a user viewing an image, and determination means for determining the size of a high-resolution region of the image to be displayed at higher resolution compared to other regions on the basis of the information acquired by the acquisition unit.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an electronic device, a control method for an electronic device, and a program. [Background technology]

[0002] In recent years, virtual reality (VR) content has become widespread along with HMD devices. To increase the appeal to users, there are attempts to display VR content in high definition (high resolution). This increases the rendering capabilities required of HMD devices, and the processing load of HMD devices tends to increase. There is therefore technology that efficiently reduces the processing load of HMD devices by adaptively increasing or decreasing the processing load of HMD devices according to the user's state. In addition, there has long been technology that changes the display form of VR content according to the user's line of sight. These technologies require appropriate display according to the user's state.

[0003] Patent Document 1 discloses a method for changing the display form of an advertisement when the user's gaze point is detected in an area that is likely to attract the user's attention. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-86302 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology disclosed in Patent Document 1, for example, when the viewpoint is not detected in an area likely to attract the user's attention, the processing load of the HMD device cannot be reduced. Also, there are cases where the viewpoint is not detected in an area likely to attract the user's attention, but the user wants to concentrate. In this case, if an advertisement is displayed, it may be difficult for the user to concentrate, and convenience may be impaired.

[0006] An object of the present invention is to provide a technique capable of efficiently reducing the processing load of a device without impairing user convenience. [Means for solving the problem]

[0007] A first aspect of the present invention is an electronic device characterized by having an acquisition means for acquiring information regarding the eye condition of a user viewing an image, and a determination means for determining, based on the information acquired by the acquisition means, the size of a high-resolution area, which is an area in the image to be displayed at a higher resolution than other areas.

[0008] A second aspect of the present invention is a method for controlling an electronic device, characterized by comprising an acquisition step of acquiring information regarding the eye condition of a user viewing an image, and a determination step of determining, based on the information acquired by the acquisition step, the size of a high-resolution area, which is an area in the image to be displayed at a higher resolution than other areas.

[0009] A third aspect of the present invention is a program for causing a computer to function as each of the means of the electronic device described above. Effect of the Invention

[0010] According to the present invention, the processing load of a device can be efficiently reduced without impairing user convenience. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an external view of a head mounted display (HMD). [Diagram 2] FIG. 2 is a block diagram showing the internal configuration of the HMD. [Diagram 3] 1 is an explanatory diagram of the principle of a gaze detection method. [Figure 4] FIG. 2 shows an eye image. [Diagram 5] 13 is a flowchart of a gaze detection operation. [Figure 6]1 is a flowchart showing a region determination process according to the first embodiment. [Figure 7] 11 is a flowchart showing a process of acquiring a fatigue level based on eyelid opening. [Figure 8] FIG. 13 is a diagram illustrating the degree of fatigue based on eyelid opening. [Figure 9] 13 is a flowchart showing a fatigue level acquisition process based on a moving range of a viewpoint. [Figure 10] 11 is a diagram illustrating a fatigue level based on a moving range of a viewpoint. [Figure 11] 13 is a flowchart showing a fatigue level acquisition process based on a variation in viewpoint. [Figure 12] FIG. 11 is a diagram illustrating a fatigue level based on a variation in viewpoint. [Figure 13] FIG. 1 is a diagram illustrating a human visual field. [Figure 14] 13 is a flowchart showing a region determination process according to the second embodiment. [Figure 15] 13 is an example of viewpoint-based resizing of a high-resolution region. [Figure 16] 13 is a flowchart showing a region determination process according to the third embodiment. [Figure 17] 13 is an example of changing the size or shape of a high resolution region based on viewpoint. [Figure 18] 13 is a flowchart showing a region determination process according to the fourth embodiment. [Figure 19] FIG. 13 is an explanatory diagram of the distribution of viewpoints. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <<Example 1>> <Configuration Description> 1(a) and 1(b) are external views of a head mounted display 100 (HMD100) according to a first embodiment. The HMD100 is an example of an electronic device. As shown in FIG. 1(a), the HMD100 is provided with a headband 200. A user brings the HMD100 into contact with his / her eyes and fixes it to his / her head with the headband 200. The user aligns his / her left eye with the left eyepiece 102a and peers into a left-eye display (not shown) from the left eyepiece 102a with the left eye, and aligns his / her right eye with the right eyepiece 102b and peers into a right-eye display (not shown) from the right eyepiece 102b with the right eye.

[0013] 2 is a block diagram showing the internal configuration of the HMD 100. The left eye 101a is the left eye of the user, and the right eye 101b is the right eye of the user.

[0014] The left eyepiece 103a and the right eyepiece 103b are lenses for adjusting the focus of the image viewed by the user.

[0015] The left light splitter 104a transmits light from the left eye display 107a (image displayed on the left eye display 107a) and guides it to the left eyepiece 103a. The left light splitter 104a also reflects light from the left eyepiece 103a and left eyepiece unit 102a (subject light reflected by the left eye 101a and indicating the left eye 101a) and guides it to the left light receiving lens 105a. The right light splitter 104b transmits light from the right eye display 107b (image displayed on the right eye display 107b) and guides it to the right eyepiece 103b. The right light splitter 104b also reflects light from the right eyepiece 103b and right eyepiece unit 102b (subject light reflected by the right eye 101b and indicating the right eye 101b) and guides it to the right light receiving lens 105b.

[0016] The left light receiving lens 105a guides the light from the left light splitter 104a to the left imaging element 106a, and the right light receiving lens 105b guides the light from the right light splitter 104b to the right imaging element 106b.

[0017] The left imaging element 106a photoelectrically converts the light incident from the left light receiving lens 105a. This captures an image of the left eye 101a. The right imaging element 106b photoelectrically converts the light incident from the right light receiving lens 105b. This captures an image of the right eye 101b. The image data of the left eye 101a and the image data of the right eye 101b are sent to the necessary blocks via bus 109.

[0018] The left-eye display 107a and the right-eye display 107b display images of virtual objects. The HMD 100 can perform stereoscopic display in which a virtual object is arranged in a three-dimensional space (a three-dimensional space centered on the HMD 100 (a user wearing the HMD 100)) using the right-eye display and the left-eye display.

[0019] The bus 109 is a bus for enabling data transmission and reception between blocks, and enables data transmission and reception between blocks connected to the bus 109.

[0020] The CPU 110 controls each block of the HMD 100 and performs various processes of the HMD 100. For example, the CPU 110 can detect the line of sight direction and line of sight position of the user by using images (images of the user's eyes) captured by the left imaging element 106a and the right imaging element 106b.

[0021] The ROM 111 stores in advance programs for processing executed by the CPU 110, information necessary for the processing, and the like. The RAM 112 stores data of images captured by the left imaging element 106a and the right imaging element 106b. The RAM 112 is also used as a work memory for temporarily storing data necessary for the processing of the CPU 110.

[0022] The illumination light source drive circuit 113 controls the lighting and light amount of the left light sources 13a1 and 13b1 that illuminate the left eye 101a, and controls the lighting and light amount of the right light sources 13a2 and 13b2 that illuminate the right eye 101b.

[0023] <Gaze detection operation> The gaze detection method will be described with reference to Figs. 3, 4(a), 4(b), and 5. Fig. 3 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection. As shown in Fig. 3, the light sources 13a and 13b are disposed approximately symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the user's eyeball 14. The light sources 13a and 13b are the left light sources 13a1 and 13b1 or the right light sources 13a2 and 13b2 in Fig. 2, the light receiving lens 16 is the left light receiving lens 105a or the right light receiving lens 105b, and the eyeball 14 is the left eye 101a or the right eye 101b. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is collected by the light receiving lens 16 on the eye imaging element 17. The eye imaging element 17 is the left imaging element 106a or the right imaging element 106b. Fig. 4(a) is a schematic diagram of an eye image captured by the eye imaging element 17 (eyeball image projected onto the eye imaging element 17), and Fig. 4(b) is a diagram showing the output intensity of the CCD in the eye imaging element 17. Fig. 5 is a schematic flowchart of the gaze detection operation.

[0024] 5, when the gaze detection operation starts, CPU 110 drives light sources 13a and 13b to emit infrared light toward user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on eye imaging element 17 through light receiving lens 16 and photoelectrically converted by eye imaging element 17. As a result, an electrical signal of the eye image that can be processed is obtained.

[0025] In step S502, the CPU 110 receives an eye image (eye image signal; The eye image (electrical signal) is acquired.

[0026] In step S503, the CPU 110 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S502.

[0027] The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are collected by the light receiving lens 16 and focused on the ocular imaging element 17 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from the ends a and b of the pupil 141 are also focused on the ocular imaging element 17 to become pupil end images a' and b' in the eye image.

[0028] FIG. 4(b) shows luminance information (luminance distribution) of the region α in the eye image of FIG. 4(a). In FIG. 4(b), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In this embodiment, the coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb. As shown in FIG. 4(b), an extremely high level of luminance is obtained at the coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the ocular imaging element 17), an extremely low level of luminance is obtained except for the coordinates Xd, Xe. And, in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143), a luminance intermediate between the above two types of luminance is obtained. Specifically, in the region where the X coordinate (coordinate in the X-axis direction) is smaller than coordinate Xa, and in the region where the X coordinate is larger than coordinate Xb, a luminance intermediate between the above two types of luminance is obtained.

[0029] From the luminance distribution as shown in FIG. 4(b), the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd', Pe' can be obtained as the coordinates of the corneal reflection images, and the coordinates of the pupil edge images a', b' can be obtained as the coordinates of the pupil edge images a', b'. In addition, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of the light beam from the pupil center c on the ocular imaging element 17 can be expressed as Xc≒(Xa+Xb) / 2. In other words, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this manner, the coordinates of the corneal reflection images Pd', Pe' and the coordinates of the pupil center image c' can be estimated.

[0030] In step S504, CPU 110 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 14 relative to light receiving lens 16, and can be found using a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.

[0031] In step S505, CPU 110 calculates the rotation angle of the optical axis of eyeball 14 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of cornea 142 approximately coincide with each other. For this reason, if the standard distance from the center of curvature O of cornea 142 to the pupil center c of pupil 141 is Oc, then the rotation angle θx of eyeball 14 in the ZX plane (plane perpendicular to the Y axis) can be calculated by the following Equation 1. The rotation angle θy of eyeball 14 in the ZY plane (plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0032] In step S506, the CPU 110 uses the rotation angles θx and θy calculated in step S505 to calculate the user's viewpoint (the position where the user's gaze is fixed; If the gaze position (coordinates of the viewpoint) (Hx, Hy) corresponds to the coordinates of the pupil center c, the gaze position (Hx, Hy) can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0033] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of camera 1, and is a conversion coefficient that converts rotation angles θx, θy into coordinates corresponding to the pupil center c on the display surface (screen) of HMD 100. Parameter m is determined in advance and stored in memory unit 129. Parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in gaze, and are acquired by performing a calibration operation. Parameters Ax, Bx, Ay, and By are stored in memory unit 129 before the gaze detection operation starts.

[0034] In step S507, CPU 110 stores the gaze position (Hx, Hy) in memory unit 129, and ends the gaze detection operation.

[0035] <Area determination process> 6 is a flowchart showing a region determination process according to Example 1. When a command to display a virtual object at a high resolution (a resolution at which an image is displayed in fine detail) is issued in the video display process in the HMD 100, the region determination process is started.

[0036] In step S601, the CPU 110 performs the above-mentioned gaze detection operation. The CPU 110 captures an image of the user's eyes (eyeball image) viewing an image with the left imaging element 106a and the right imaging element 106b, and acquires (stores) the image as eye information (eyeball information). The eye information includes information on the user's viewpoint.

[0037] In step S602, the CPU 110 acquires information related to the state of the user's eyes. In the first embodiment, the information related to the state of the user's eyes is information related to the user's fatigue level. The CPU 110 acquires the user's fatigue level through a fatigue level acquisition process. The fatigue level acquisition process will be described later with reference to Figs. 7 to 12(b).

[0038] In step S603, CPU 110 determines the size (size, area) of the high resolution region based on the fatigue level. The high resolution region is a region that is displayed at a higher resolution than other regions in the image viewed by the user. The relationship between the fatigue level and the size of the high resolution region will be described later with reference to FIG. 8(c). After determining the size of the high resolution region, CPU 110 ends the region determination process.

[0039] <Fatigue level acquisition process> Three examples of fatigue level acquisition processes executed in step S602 in Fig. 6 are shown below. The fatigue level may be a value acquired by one of these processes, or a value that is a combination of values ​​acquired by two or more processes.

[0040] FIG. 7 is a flowchart showing a process for obtaining a fatigue level based on eyelid opening.

[0041] In step S701, CPU 110 acquires information on eyelid opening (measures eyelid opening) based on eye information (eye image).

[0042] In step S702, the CPU 110 acquires the fatigue level based on the eyelid opening. The relationship between the eyelid opening and the fatigue level is shown in Fig. 8(b). After acquiring the fatigue level, the CPU 110 ends the fatigue level acquisition process.

[0043] Fig. 8(a) is a diagram showing the distance between the upper and lower eyelids of a user. In step S701 in Fig. 7, CPU 110 acquires (measures) the distance between the upper and lower eyelids to acquire information on the opening of the eyelids. The method for acquiring the distance between the upper and lower eyelids may be a known method, and for example, the eyeball region in the eye image may be specified by semantic region division, and the height of the eyeball region (the width in the vertical direction) may be measured.

[0044] Depending on the user, the distance between the upper and lower eyelids when the degree of fatigue is high (for example, the average value per unit time) is smaller than the distance between the upper and lower eyelids when the degree of fatigue is low. Therefore, in step S702 in Fig. 7, when the distance between the upper and lower eyelids is large, CPU 110 acquires a smaller value as the degree of fatigue than when the distance is small. In the example in Fig. 8(b), CPU 110 acquires a smaller value as the distance between the upper and lower eyelids is large, and acquires a larger value as the distance is small.

[0045] Depending on the user, the movement amount of the viewpoint when the fatigue level is high is smaller than the movement amount of the viewpoint when the fatigue level is low. Therefore, in step S603 in FIG. 6, when the fatigue level is high, the CPU 110 determines the size of the high resolution region to be smaller than when the fatigue level is low. In the example of FIG. 8(c), the CPU 110 determines the size of the high resolution region to be smaller as the fatigue level is high and larger as the fatigue level is low. When it is expected that the user is highly tired based on the results acquired by the fatigue level acquisition process, the processing load of the HMD 100 can be reduced without impairing convenience by making the size of the high resolution region smaller than normal.

[0046] Here, CPU 110 may acquire the degree of fatigue not only based on the opening of the eyelids but also based on the user's viewpoint, etc. For example, CPU 110 may acquire the degree of fatigue based on the movement range of the viewpoint or the variation in the viewpoint.

[0047] FIG. 9 is a flowchart showing a fatigue level acquisition process based on the moving range of the viewpoint.

[0048] In step S901, CPU 110 acquires information on the movement range of the viewpoint based on the eye information (measures the size of the movement range of the viewpoint).

[0049] In step S902, CPU 110 acquires a fatigue level based on the size of the movement range of the viewpoint. The relationship between the size of the movement range of the viewpoint and the fatigue level is shown in Fig. 10(b). After acquiring the fatigue level, CPU 110 ends the fatigue level acquisition process.

[0050] 10(a) is a diagram showing the movement range of the viewpoint. For example, CPU 110 acquires the size (area) of a rectangular range defined by the range from the right end to the left end and the range from the top end to the bottom end of the viewpoints detected per unit time as the movement range of the viewpoint. Note that CPU 110 may acquire the range (width) from the right end to the left end or the range from the top end to the bottom end of the viewpoints detected per unit time as the movement range of the viewpoint.

[0051] Depending on the user, the size of the range of movement of the viewpoint when the degree of fatigue is large may be smaller than the size of the range of movement of the viewpoint when the degree of fatigue is small. Thus, in step S902 of FIG. 9, when the size of the range of movement of the viewpoint is large, CPU 110 acquires a smaller value as the degree of fatigue than when the size is small. In the example of FIG. 10(b), CPU 110 acquires a smaller value as the size (area) of the range of movement of the viewpoint is larger, and acquires a larger value as the size is smaller. In addition, when the fatigue level is acquired based on the size of the moving range of the viewpoint, the relationship between the fatigue level and the high-resolution area may be the same as that shown in the graph of FIG.

[0052] In this way, by obtaining the degree of fatigue based on the size of the range of movement of the viewpoint and determining the size of the high-luminance area based on the degree of fatigue, it is possible to reduce the processing load of the HMD 100 without compromising convenience.

[0053] FIG. 11 is a flowchart showing a fatigue level acquisition process based on the variation in the viewpoint (gazing point).

[0054] In step S1101, CPU 110 acquires information on the variation in viewpoint based on eye information (measures the variation in viewpoint).

[0055] In step S1102, CPU 110 acquires a fatigue level based on the variation in viewpoint. The relationship between the variation in viewpoint and the fatigue level is shown in Fig. 12(b). After acquiring the fatigue level, CPU 110 ends the fatigue level acquisition process.

[0056] 12A is a diagram showing the variation of the viewpoint. For example, the CPU 110 obtains the variation of the viewpoint by summing the variance value in the X direction and the variance value in the Y direction of the viewpoint detected per unit time.

[0057] Depending on the user, the viewpoints detected per unit time are concentrated in a part (for example, the center) when the degree of fatigue is high, and are evenly distributed when the degree of fatigue is low. That is, the variation in the viewpoint when the degree of fatigue is high is smaller than the variation in the viewpoint when the degree of fatigue is low. Therefore, in step S1102 of FIG. 11, when the variation in the viewpoint is large, the CPU 110 acquires a smaller value as the degree of fatigue than when the variation in the viewpoint is small. In the example of FIG. 12(b), the CPU 110 acquires a smaller value as the variation in the viewpoint (variance value) is large, and a larger value as the variation in the viewpoint is small. Note that the relationship between the degree of fatigue and the high-resolution area when the degree of fatigue is acquired based on the variation in the viewpoint may be the same as the graph of FIG. 8(c).

[0058] In this way, by acquiring the degree of fatigue based on the variation in the viewpoint and determining the size of the high-luminance area based on the degree of fatigue, it is possible to reduce the processing load of the HMD 100 without compromising usability.

[0059] The method of acquiring the fatigue level is not limited to the examples shown in Figs. 7 to 12(b). The magnitude relationship between the value acquired from the eye information and the fatigue level may be the opposite of that in the above-mentioned example. For example, in step S702 in Fig. 7, the CPU 110 may acquire a larger value as the fatigue level when the distance between the upper and lower eyelids is large than when the distance is small. Also, for example, a two-level value may be acquired as the fatigue level.

[0060] The human visual field and the high-resolution region will be described in detail with reference to Fig. 13. Fig. 13 is a diagram showing the human visual field. The human visual field includes peripheral and central visual fields, and the appearance differs between the peripheral and central visual fields. The peripheral visual field is a range in which objects and information within the visual field can be recognized or distinguished and viewed in detail, and the central visual field is a range in which objects and information within the visual field can be consciously recognized. The peripheral visual field is located outside the central visual field.

[0061] The CPU 110 calculates a radius r from the user's viewpoint (point of gaze, center of multiple viewpoints) as the origin so that the central visual field is displayed with a higher resolution than the peripheral visual field. 1 In addition, the system determines a high-resolution region 161 based on the user's viewpoint so that the peripheral vision is displayed at a lower resolution than the central vision. Point with radius r 2 From the area of ​​radius r 1 The low-resolution area excluding the area (high-resolution area 161) The central visual field is displayed at a high resolution, which does not impair user convenience. Also, the peripheral visual field is displayed at a low resolution, which reduces the load on the rendering process of the HMD 100 and allows for efficient use of computational resources.

[0062] Based on the user's degree of fatigue, the CPU 110 determines the sizes of the high resolution region 161 and the low resolution region 162. For example, when the degree of fatigue is high, the CPU 110 makes the size of the high resolution region 161 (proportion to the entire image region) smaller and the size of the low resolution region 162 larger than when the degree of fatigue is low.

[0063] In FIG. 13, high resolution area 161 and low resolution area 162 are circular, but CPU 110 may change the shape of each area depending on the user's degree of fatigue. For example, it is considered that when the degree of fatigue is high, the amount of horizontal gaze movement is larger than when the degree of fatigue is low. Therefore, CPU 110 may change the shape of the high resolution area to a shape whose horizontal length is longer than its vertical length. Also, CPU 110 may change the shape of the high resolution area to a shape whose horizontal length is longer when the degree of fatigue is high than when the degree of fatigue is low.

[0064] In this way, by determining the size of the high-resolution area based on the user's degree of fatigue, the processing load of the HMD 100 can be reduced without compromising convenience.

[0065] <<Example 2>> The CPU 110 determines the size of the high resolution region based on the user's fatigue level in the first embodiment, but determines the size of the high resolution region based on the user's viewpoint in the second embodiment. The following mainly describes the differences from the first embodiment.

[0066] <Area determination process> FIG. 14 is a flowchart illustrating the region determination process according to the second embodiment.

[0067] In step S1401, CPU 110 performs a gaze detection operation to obtain information about the user's eyes. The process in step S1401 is the same as step S601 in FIG.

[0068] In step S1402, CPU 110 acquires information relating to the movement range of the viewpoint (information on the movement range of the viewpoint) from the eye information.

[0069] In step S1403, CPU 110 determines the size of the high resolution region based on information related to the movement range of the viewpoint, and ends the region determination process. It is assumed that the shape of the high resolution region is a circle with the viewpoint as its origin.

[0070] In the first embodiment, the degree of fatigue is obtained based on the size of the range of movement of the viewpoint, for example, but depending on the user, there may be no correlation between the size of the range of movement of the viewpoint and the degree of fatigue. Also, some people move their gaze little regardless of the degree of fatigue, while others do not change their gaze at all. Therefore, in the second embodiment, information related to the range of movement of the viewpoint is regarded as an individual characteristic, and the size of the high-resolution area is determined based on the information (based on the viewpoint).

[0071] 15(a) and 15(b), an example will be described in which information on the bias of the viewpoint relative to the center of the moving range of the viewpoint is obtained as information on the moving range of the viewpoint, and the size of the high resolution area is determined based on the information.

[0072] FIG. 15A is a diagram showing the center of gravity of the viewpoints (center of gravity of a plurality of viewpoints) when the viewpoints are evenly distributed and when the viewpoints are biased with respect to the center of the moving range of the viewpoints. For example, the center of a rectangular range defined by a range from the right end to the left end and a range from the top end to the bottom end among the viewpoints detected per unit time. When the center of gravity of the viewpoint is located near the boundary of the viewpoint movement range, the bias of the viewpoint with respect to the center of the viewpoint movement range (hereinafter simply referred to as viewpoint bias) is larger than when the center of gravity of the viewpoint is located near the center of the viewpoint movement range. When the viewpoint is biased, the user does not gaze evenly over the range of line of sight movement, and there is a high possibility that the movement direction of the viewpoint will suddenly change, so it is desirable to set the high resolution area large. When the viewpoints are evenly distributed, the user gazes evenly over the range of the viewpoint movement, and there is a low possibility that the movement direction of the viewpoint will suddenly change, so there is no need to set the high resolution area large.

[0073] Therefore, the CPU 110 may obtain information on the bias of the viewpoint as information on the movement range of the viewpoint, and determine the size of the high resolution region based on the bias of the viewpoint. For example, in step S1402 in FIG. 14, the CPU 110 obtains the distance from the center of the movement range of the viewpoint to the center of gravity of the viewpoint as information on the bias of the viewpoint. In the example of FIG. 15(a), the bias of the viewpoint is smaller as the distance from the center of the movement range of the viewpoint to the center of gravity of the viewpoint is smaller. In step S1403 in FIG. 14, when the bias of the viewpoint is small, the CPU 110 determines a size (radius) smaller than when the bias of the viewpoint is large as the size of the high resolution region. FIG. 15(b) is a diagram showing the relationship between the distance from the center of the movement range of the viewpoint to the center of gravity of the viewpoint and the radius of the circle of the high resolution region. In the example of FIG. 15(b), the CPU 110 determines a size smaller as the bias of the viewpoint (the distance from the center of the movement range of the viewpoint to the center of gravity of the viewpoint) is smaller, and determines a size larger as the bias of the viewpoint is larger as the bias of the viewpoint is larger. This reduces the processing load on the HMD 100. In addition, for example, when the direction of movement of the viewpoint suddenly changes, high-resolution display is also performed at the new location, so user convenience is not compromised.

[0074] With reference to FIGS. 15(c) and 15(d), an example will be described in which information on the variation (variance value) of the viewpoint is obtained as information on the moving range of the viewpoint, and the size of the high resolution region is determined based on this information.

[0075] 15(c) is a diagram showing the cases where the viewpoints are evenly distributed within the range of viewpoint movement and where the viewpoints are concentrated in the center. Compared to the case where the viewpoints are evenly distributed (large variance value), when the viewpoints are highly biased, such as when the viewpoints are concentrated in the center (small variance value), it is desirable to set a large high-resolution area. For example, this is to ensure that when the user moves the viewpoint to an area different from the area where it is currently concentrated, the image will be displayed in high resolution at the new location.

[0076] Therefore, the CPU 110 may obtain information on the variation of the viewpoint as information on the movement range of the viewpoint, and determine the size of the high-resolution region based on the variation of the viewpoint. For example, in step S1402 of FIG. 14, the CPU 110 obtains the sum of the variance value in the X direction and the variance value in the Y direction of the viewpoint detected per unit time as the information on the variation of the viewpoint. In step S1403, when the variation of the viewpoint is large, the CPU 110 determines the size (radius) of the high-resolution region to be smaller than when the variation of the viewpoint is small. FIG. 15(d) is a diagram showing the relationship between the variance value and the radius of the circle of the high-resolution region. In the example of FIG. 15(d), the CPU 110 determines the size of the high-resolution region to be smaller as the variation (variance value) of the viewpoint is larger, and larger as the variation of the viewpoint is smaller. This reduces the processing load of the HMD 100. In addition, for example, when the movement direction of the viewpoint suddenly changes, high resolution display is performed at the destination, so that the convenience of the user is not impaired.

[0077] The relationship between the bias of the viewpoint and the size of the high resolution region may be reversed, or the relationship between the variation of the viewpoint and the size of the high resolution region may be reversed. Furthermore, the CPU 110 may determine the size of the high resolution region according to characteristics of the viewpoint other than those described above. For example, the CPU 110 may determine the size of the high resolution region based on the size of the movement range of the user's viewpoint. When the movement range of the viewpoint is small, CPU 110 may determine the size of the high resolution region to be smaller than when the movement range of the viewpoint is large.

[0078] <<Example 3>> The CPU 110 determines the size of the high resolution region based on the user's fatigue level in the first embodiment, but determines the size or shape of the high resolution region based on the user's viewpoint in the third embodiment. The following mainly describes the differences from the first embodiment.

[0079] <Area determination process> FIG. 16 is a flowchart illustrating the area determination process according to the third embodiment.

[0080] In step S1601, CPU 110 performs a gaze detection operation to obtain information about the user's eyes. The process in step S1601 is the same as step S601 in FIG.

[0081] In step S1602, CPU 110 acquires the distance (viewpoint distance) from the center of the display unit (left eye display 107a or right eye display 107b) to the viewpoint based on the eye information. The center of the display unit may be interpreted as the center of the image the user is looking at.

[0082] In step S1603, CPU 110 determines the size (or shape) of the high resolution area based on the viewpoint distance, and ends the area determination process.

[0083] Here, the further the user's viewpoint is from the center of the display unit, the smaller the amount of movement of the viewpoint becomes. For example, the video content viewed by the user is often displayed over the entire display unit, and various operation items are often displayed at the edge of the display unit. When the user's viewpoint is at the center of the display unit, the user is trying to view the video content, and the amount of movement of the viewpoint is often large in order to view the entire video content. When the user's viewpoint is at the edge of the display unit, the user is trying to view the operation items, and the amount of movement of the viewpoint is often small. Therefore, in step S1603, when the user's viewpoint is farther from the center of the display unit, CPU 110 may determine a size of the high image area that is smaller than when the user's viewpoint is close to the center of the display unit.

[0084] Fig. 17(a) is a diagram showing an example of changing the size of the high-resolution area according to the viewpoint distance. In Fig. 17(a), the size of the high-resolution area is smaller when the viewpoint is at a position that is a predetermined distance or more away from the center of the display unit than when the viewpoint is at the center of the display unit. The size of the high-resolution area may be smaller the closer the viewpoint is to an edge of the display unit (for example, near one of the four corners). Note that in Fig. 17(a), the shape of the high-resolution area is a circle with the user's viewpoint as the origin, regardless of the viewpoint distance.

[0085] FIG. 17(b) is a diagram showing an example of changing the shape of the high-resolution area according to the viewpoint distance. When the user's viewpoint is at the center of the display unit, the shape of the high-resolution area is a circle with the viewpoint as the origin, but when the viewpoint is at a position away from the center of the display unit by a predetermined distance or more, the shape is an ellipse extending in a direction passing through the viewpoint and the center of the display unit. When the viewpoint is further away from the center of the display unit, the high-resolution area is set with the center being a position closer to the center of the display unit than the viewpoint. In the example of FIG. 17(b), when the viewpoint is at a position away from the center of the display unit, the viewpoint is at the center of the ellipse, but when the viewpoint is at the edge, the viewpoint is closer to the edge of the display unit than the center of the ellipse. In other words, the spread of the high-resolution area on the side closer to the viewpoint from the center of the display unit is larger than the spread of the high-resolution area on the side farther from the center of the display unit. The shape and position of the high-resolution area are not limited to these, and may be determined to include the user's viewpoint.

[0086] An example of changing the size of the high-resolution region is shown in Fig. 17(a), and a sample is shown in Fig. 17(b). In the above, an example of changing the size, shape, and position (position relative to the viewpoint) has been described. However, the present invention is not limited to this, and at least one of the size, shape, and position may be changed. For example, the CPU 110 may change only the shape, or may change only the position relative to the viewpoint. In addition, in Figs. 17(a) and 17(b), the dashed lines drawn for the case where the viewpoint is located away from the center of the display unit and the case where the viewpoint is located at the edge of the display unit indicate the high resolution area when the viewpoint is located at the center of the display unit, and are not actually displayed.

[0087] In this way, by determining the size or shape of the high-resolution area based on the user's viewpoint, the processing load of the HMD 100 can be reduced without compromising usability.

[0088] <<Example 4>> In the first embodiment, the CPU 110 determines the size of the high-resolution region based on the degree of fatigue of the user, but in the fourth embodiment, the CPU 110 determines the size of the high-resolution region based on the probability that the user's gaze point is detected in each region. The following mainly describes the differences from the first embodiment.

[0089] <Area determination process> FIG. 18 is a flowchart illustrating the region determination process according to the fourth embodiment.

[0090] In step S1801, CPU 110 performs a gaze detection operation to obtain information about the user's eyes. The process in step S1801 is the same as step S601 in FIG.

[0091] In step S1802, CPU 110 acquires past eye information of the user.

[0092] In step S1803, CPU 110 acquires the probability that the user's gaze point will be detected in each area (gaze point detection probability, frequency) from past eye information.

[0093] In step S1804, CPU 110 determines the size of the high-resolution region based on the user's current viewpoint and the viewpoint detection probability. Then, CPU 110 ends the region determination process. Note that the shape of the high-resolution region is a circle with the viewpoint as the origin.

[0094] FIG. 19(a) is a diagram showing the distribution of detected viewpoints. FIG. 19(b) is a diagram showing the sum of the viewpoints detected in each region. As shown in FIGS. 19(a) and 19(b), each region is divided concentrically with the center of the display unit (left-eye display 107a or right-eye display 107b) as the origin. As shown in FIG. 19(b), the number of detected viewpoints tends to decrease as it moves away from the center of the display unit. Therefore, when the current viewpoint is included in a location (region) where the number of past viewpoints detected is large, the CPU 110 widens the high-resolution region, and when the current viewpoint is included in a location where the number of past viewpoints detected is small, the CPU 110 narrows the high-resolution region. That is, when the user's current viewpoint is included in a region where the viewpoint detection probability is low, the CPU 110 determines the size of the high-resolution region to be smaller than when the user's current viewpoint is included in a region where the viewpoint detection probability is high. For example, the CPU 110 determines the size of the high-resolution region to be smaller the lower the viewpoint detection probability corresponding to the current viewpoint is, and larger the higher the viewpoint detection probability corresponding to the current viewpoint is.

[0095] CPU 110 may change the shape of the high-resolution region based on the probability distribution of the gaze detection. For example, CPU 110 may obtain the amount of deviation of the point with the highest gaze detection probability from the center of the display unit. Then, when the amount of deviation is greater than a threshold, CPU 110 may determine the shape of the high-resolution region to be an ellipse extending in a direction passing through the point with the highest gaze detection probability and the center of the display unit.

[0096] In this way, the size of the high-resolution regions is calculated based on the probability of detecting the user's gaze in each region. By determining the size of the display, the processing load of the HMD 100 can be reduced without impairing usability.

[0097] The above embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above embodiment.

[0098] <Other Examples> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0099] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An acquisition means for acquiring information regarding an eye condition of a user viewing an image; a determination means for determining a size of a high-resolution region, which is a region in the image to be displayed at a higher resolution than other regions, based on the information acquired by the acquisition means; have 1. An electronic device comprising: (Configuration 2) the information is information regarding a fatigue level of the user, The determining means determines a size of the high resolution region based on the degree of fatigue. 2. The electronic device according to configuration 1. (Configuration 3) The determining means determines, when the degree of fatigue is high, a size smaller than that when the degree of fatigue is low, as the size of the high resolution region. 3. The electronic device according to configuration 2. (Configuration 4) The acquiring means acquires the fatigue level based on an image of the user's eyes. 4. The electronic device according to configuration 2 or 3. (Configuration 5) The acquisition means includes: obtaining information on an opening of the user's eyelids, a range of movement of the user's gaze point, or a variance in the user's gaze point from the image of the eye; The fatigue level is acquired based on the information. 5. The electronic device according to configuration 4. (Configuration 6) the information being information regarding a viewpoint of the user, The determining means determines a size of the high resolution region based on a viewpoint of the user. 2. The electronic device according to configuration 1. (Configuration 7) The determining means determines, when the user's viewpoint is far from the center of the image, a size smaller than when the user's viewpoint is close to the center of the image, as the size of the high resolution region. 7. The electronic device according to configuration 6. (Configuration 8) The determining means determines a size of the high resolution area based on a size of a moving range of the user's viewpoint. 7. The electronic device according to configuration 6. (Configuration 9) The determining means determines, when the movement range is small, a size of the high resolution region that is smaller than a size of the high resolution region that is larger than a size of the high resolution region that is larger than a size of the high resolution region. 9. The electronic device according to configuration 8. (Configuration 10) The determining means determines the size of the high resolution area based on a bias of the user's viewpoint relative to a center of a moving range of the user's viewpoint. 7. The electronic device according to configuration 6. (Configuration 11) The determining means determines, when the bias of the viewpoint is small, a size smaller than that when the bias of the viewpoint is large as the size of the high resolution region. 11. The electronic device according to configuration 10. (Configuration 12) The determining means determines a size of the high resolution region based on a variance in a viewpoint of the user. 7. The electronic device according to configuration 6. (Configuration 13) The determining means determines, when the variation in the viewpoint is large, a size of the high resolution region that is smaller than .... 13. The electronic device according to configuration 12. (Configuration 14) The determining means determines the size of the high resolution regions based on a current viewpoint of the user and a probability that the viewpoint of the user is detected in each region. 7. The electronic device according to configuration 6. (Configuration 15) The determination means determines, when the low probability region includes the current viewpoint of the user, a size smaller than a size when the high probability region includes the current viewpoint of the user, as the size of the high resolution region. 15. The electronic device according to configuration 14. (Configuration 16) The determining means determines a ratio of the high resolution area to the entire area of ​​the image. 16. The electronic device according to any one of configurations 1 to 15. (Configuration 17) The determining means determines a size and a shape of the high resolution region based on the information. 17. The electronic device according to any one of configurations 1 to 16. (Configuration 18) The determining means determines the high resolution region that includes the user's viewpoint. 18. An electronic device according to any one of configurations 1 to 17. (Configuration 19) The electronic device is a head-mounted display. 19. The electronic device according to any one of configurations 1 to 18. (method) An acquisition step for acquiring information regarding an eye condition of a user viewing the image; a determination step of determining a size of a high resolution region, which is a region in the image to be displayed at a higher resolution than other regions, based on the information acquired by the acquisition step; have 23. A method for controlling an electronic device comprising: (program) A computer is configured to function as each of the means of the information processing device according to any one of configurations 1 to 19. A program to make this possible. [Explanation of symbols]

[0100] 100: HMD 110: CPU

Claims

1. An acquisition means for acquiring information regarding an eye condition of a user viewing an image; a determination means for determining a size of a high-resolution region, which is a region in the image to be displayed at a higher resolution than other regions, based on the information acquired by the acquisition means; have 1. An electronic device comprising:

2. the information is information regarding a fatigue level of the user, The determining means determines a size of the high resolution region based on the degree of fatigue.

2. The electronic device according to claim 1 .

3. The determining means determines, when the degree of fatigue is high, a size smaller than that when the degree of fatigue is low, as the size of the high resolution region.

3. The electronic device according to claim 2.

4. The acquiring means acquires the fatigue level based on an image of the user's eyes.

3. The electronic device according to claim 2.

5. The acquisition means includes: obtaining information on an opening of the user's eyelids, a range of movement of the user's gaze point, or a variance in the user's gaze point from the image of the eye; The fatigue level is acquired based on the information.

5. The electronic device according to claim 4.

6. the information being information regarding a viewpoint of the user, The determining means determines a size of the high resolution region based on a viewpoint of the user.

2. The electronic device according to claim 1 .

7. The determining means determines, when the user's viewpoint is far from the center of the image, a size smaller than when the user's viewpoint is close to the center of the image, as the size of the high resolution region.

7. The electronic device according to claim 6.

8. The determining means determines a size of the high resolution area based on a size of a moving range of the user's viewpoint.

7. The electronic device according to claim 6.

9. The determining means determines, when the movement range is small, a size of the high resolution region that is smaller than a size of the high resolution region that is larger than a size of the high resolution region that is larger than a size of the high resolution region.

9. The electronic device according to claim 8.

10. The determining means determines the size of the high resolution area based on a bias of the user's viewpoint relative to a center of a moving range of the user's viewpoint.

7. The electronic device according to claim 6.

11. The determining means determines, when the bias of the viewpoint is small, a size smaller than that when the bias of the viewpoint is large, as the size of the high resolution region.

11. The electronic device according to claim 10.

12. The determining means determines a size of the high resolution region based on a variance in a viewpoint of the user.

7. The electronic device according to claim 6.

13. The determining means determines, when the variation in the viewpoint is large, a size of the high resolution region that is smaller than ....

13. The electronic device according to claim 12.

14. The determining means determines the size of the high resolution regions based on a current viewpoint of the user and a probability that the viewpoint of the user is detected in each region.

7. The electronic device according to claim 6.

15. The determination means determines, when the low probability region includes the current viewpoint of the user, a size smaller than a size when the high probability region includes the current viewpoint of the user, as the size of the high resolution region.

15. The electronic device according to claim 14.

16. The determining means determines a ratio of the high resolution area to the entire area of ​​the image.

2. The electronic device according to claim 1 .

17. The determining means determines a size and a shape of the high resolution region based on the information.

2. The electronic device according to claim 1 .

18. The determining means determines the high resolution region that includes the user's viewpoint.

2. The electronic device according to claim 1 .

19. The electronic device is a head-mounted display.

2. The electronic device according to claim 1 .

20. An acquisition step for acquiring information regarding an eye condition of a user viewing the image; a determination step of determining a size of a high resolution region, which is a region in the image to be displayed at a higher resolution than other regions, based on the information acquired by the acquisition step; have 23. A method for controlling an electronic device comprising:

21. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 19.

Images