Image processing apparatus, method, program, and storage medium

Abstract

The invention provides an image processing apparatus capable of performing appropriate rendering processing according to the overhead degree of a user. The image processing apparatus for processing an image to be displayed on a display section includes: a line-of-sight detection section for detecting a direction of a line of sight of a user viewing the display section; a first distance detection means for detecting a first distance, which is a distance to a position at which the user is watching, based on the detected direction of the line of sight; a second distance detection means for detecting a second distance, which is a distance to the subject; an obtaining means for obtaining, based on the first distance and the second distance, a bird's-eye view degree indicating a degree to which the user is viewing the subject in a bird's-eye view manner; and a changing section configured to change at least one of a region in which the processing for changing the image quality of the image is performed and an image quality in the processing for changing the image quality of the image based on the bird's-eye view degree.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that performs rendering processing on a fixation point in a head-mounted display, a head-up display, or the like. Background Art

[0002] In recent years, attention has been focused on AR (augmented reality) and VR (virtual reality). Among them, AR is a technology for displaying an image superimposed on a real-world scene in front of a user's eyes, and VR displays a real image different from the reality in front of the user's eyes. Some types of head-mounted displays (HMDs) and vehicle head-up displays (HUDs) equipped with these technologies have a line-of-sight detection function. When the line-of-sight detection function is installed, it is possible to assist the user in easily viewing a subject within the field of view on the display surface.

[0003] For example, a technique called foveated rendering is known, which aims to further improve the sense of immersion, further reduce the rendering cost, and further reduce the transmission cost.

[0004] Patent Document 1 proposes a technique that predicts the movement of the line of sight and performs high-resolution rendering processing on a region that includes not only the point of interest but also the points to which the line of sight moves in time series.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2018-004950 Summary of the Invention

[0008] Technical Problem to be Solved by the Invention

[0009] However, in a real-world scene, a user can adopt an observation method of grasping the overall movement by viewing a subject or a scene in an overlooking manner. In the overlooking state, even if the fine shape of the subject cannot be recognized, the user further places the viewpoint away from the subject so that the entire movement can be understood. In this case, the user's line of sight does not point to the subject. Therefore, with the technique described in Patent Document 1, when the user is overlooking the scene, a specific area of interest cannot be detected.

[0010] The present invention has been made in consideration of the above problems, and provides an image processing apparatus that can perform appropriate rendering processing according to the degree of the user's overlooking viewing.

[0011] Solution to the Problem

[0012] The image processing device according to the present invention is an image processing device for processing an image to be displayed on a display component, and is characterized by including: a line-of-sight detection component for detecting the direction of the line of sight of a user viewing the display component; a first distance detection component for detecting a first distance, which is the distance to the position being gazed at by the user, based on the detected direction of the line of sight; a second distance detection component for detecting a second distance, which is the distance to a subject; an acquisition component for acquiring an overlooking degree for indicating the degree to which the user is viewing the subject in an overlooking manner based on the first distance and the second distance; and a change component for changing at least one of the following based on the overlooking degree: the area for performing processing for changing the image quality of the image and the image quality in the processing for changing the image quality of the image.

[0013] Advantages of the invention

[0014] According to the present invention, appropriate rendering processing can be performed according to the degree of the user's overlooking view.

[0015] Other features and advantages of the present invention will become apparent from the following description in conjunction with the accompanying drawings. Note that in all the drawings, the same reference numerals denote the same or similar components. Description of the drawings

[0016] The drawings incorporated in and forming a part of the specification illustrate embodiments of the present invention and, together with the specification, are used to explain the principles of the present invention.

[0017] Figure 1A FIG. is a schematic configuration diagram of a head-mounted display as a first embodiment of the image processing device according to the present invention.

[0018] Figure 1B FIG. is a block diagram showing the internal configuration of the head-mounted display.

[0019] Figure 2 FIG. is a diagram explaining the principle of the line-of-sight detection method.

[0020] Figure 3 FIG. is a diagram showing an image of the eyeball projected onto an eyeball image sensor and the output intensity on the eyeball image sensor.

[0021] Figure 4 FIG. is a flowchart showing the line-of-sight detection operation.

[0022] Figure 5A FIG. is a diagram showing an example of a method for obtaining a convergence angle and detecting an overlooking degree.

[0023] Figure 5BA diagram showing an example of a method for obtaining a convergence angle and detecting an overlooking degree.

[0024] Figure 5C A diagram showing an example of a method for obtaining a convergence angle and detecting an overlooking degree.

[0025] Figure 6 A flowchart showing an operation of changing a blur area based on an overlooking degree.

[0026] Figure 7A A diagram showing an example of changing a blur area based on an overlooking degree.

[0027] Figure 7B A diagram showing an example of changing a blur area based on an overlooking degree.

[0028] Figure 8 A flowchart showing an operation of changing the quality of a rendered image based on an overlooking degree.

[0029] Figure 9A A diagram showing an example of changing the quality of a rendered image based on an overlooking degree.

[0030] Figure 9B A diagram showing an example of changing the quality of a rendered image based on an overlooking degree.

[0031] Figure 10 A flowchart showing an operation of changing a blur area and the quality of a rendered image based on an overlooking degree.

[0032] Figure 11A A diagram showing an example of changing a blur area and the quality of a rendered image based on an overlooking degree.

[0033] Figure 11B A diagram showing an example of changing a blur area and the quality of a rendered image based on an overlooking degree.

[0034] Figure 12 A flowchart showing an operation of changing a blur area and the quality of a rendered image based on an overlooking degree and a subject movement. Detailed Description of the Invention

[0035] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. In the embodiments, multiple features are described, but the invention does not require all such features, and multiple such features can be appropriately combined. Further, in the drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions thereof are omitted.

[0036] First Embodiment

[0037] Figure 1AFIG. 0 is a diagram showing a schematic configuration of a head-mounted display (hereinafter referred to as "HMD") 100 which is a first embodiment of the image processing apparatus of the present invention. Figure 1B FIG. 1 shows a block configuration of the HMD 100.

[0038] In Figure 1A FIG. 1, the left side of the figure shows the configuration of the HMD 100 viewed from the top of the user's head, and the right side shows the configuration of the gaze detection device.

[0039] When the user wears the HMD 100 on their head, the left eye 101 and the right eye 102 can observe the real space through the left-eye display 107 and the right-eye display 108 of the see-through type, respectively. By displaying videos such as operation icons and image data on these see-through displays, the displayed videos can be superimposed on the real world that the user is viewing through the displays.

[0040] Another possible configuration is as follows. That is, when in the non-see-through mode, the non-see-through display can be used to display videos stored internally (captured moving images, game videos, etc.), and when in the see-through mode, the images captured by the left-eye camera 103 and the right-eye camera 104 are displayed. In addition, a video for synthesizing the internally stored video and the images captured by the camera can also be displayed.

[0041] The position on the display that the user is gazing at is estimated by a gaze detection operation using the left gaze imaging unit 105 and the right gaze imaging unit 106 described later. The HMD 100 also includes an operation unit 109 to which functions such as a power button and various device operations can be assigned.

[0042] Figure 1B FIG. 2 shows a block diagram showing the internal configuration of the HMD 100. The sensor unit 110 detects the orientation of the HMD 100. The control unit 111 obtains image data, gaze information, orientation information, etc. from the left-eye camera 103, the right-eye camera 104, the left gaze imaging unit 105, the right gaze imaging unit 106, and the sensor unit 110, and controls the entire HMD 100.

[0043] The control information generation unit 112 generates position / orientation information related to the HMD 100 in a three-dimensional space based on the images output from the imaging processing units 114 and 115 and the sensor information output from the sensor unit 110. The position / orientation information includes the coordinates of the HMD 100 in the three-dimensional space, the direction of the user's line of sight, the rotation angle of the HMD 100 with respect to the axis in the line-of-sight direction, and the like. The sensor unit 110 uses, for example, a gyroscope to detect information such as the direction and acceleration of the HMD 100. The memory unit 113 holds virtual objects and control information to be superimposed on the real-world scene by the CG rendering unit 121.

[0044] The imaging processing units 114 and 115 perform preset image processing on the images input from the left-eye camera 103 and the right-eye camera 104. The image processing here includes gain correction, pixel defect correction, automatic exposure correction, distortion correction, and the like. The image synthesis units 120 and 122 receive the images to be synthesized from the imaging processing units 114 and 115, and also receive CG information from the CG rendering unit 121. When the frame information from the imaging processing units 114 and 115 is consistent, the CG (computer graphics) from the CG rendering unit 121 is synthesized with the captured image, and the synthesized image is output to the image editing processing unit 125.

[0045] The left-eye line-of-sight detection unit 116 and the right-eye line-of-sight detection unit 117 detect the positions of the pupils of the user who is using the HMD 100, and supply the line-of-sight information indicating which positions on the displays 107 and 108 the user is looking at to the image editing processing unit 125 as coordinate data. The object fixation determination unit 123 calculates the fixation degree, which is the degree to which the user is fixating on an object, based on the convergence angle obtained from the line-of-sight information related to both eyes from the left-eye line-of-sight detection unit 116 and the right-eye line-of-sight detection unit 117 according to the line-of-sight vector. The overlooking degree determination unit 124 determines whether the user is viewing the video in an overlooking manner based on the convergence angle and the fixation degree, which are the determination results of the object fixation determination unit 123.

[0046] The image editing processing unit 125 performs a rendering process in which, in the images from the image synthesis units 120 and 122, the area within a predetermined area centered on the user's line-of-sight direction is not processed, and the area outside this area is blurred. That is, in the present embodiment, the rendering process includes a process of changing the image quality of the image. The blurring process can be achieved by, for example, filtering. In the present embodiment, the process of changing the rendering area and the image quality is performed according to the user's overlooking degree.

[0047] The left-eye image processing unit 118 and the right-eye image processing unit 119 are controlled to display the image processed by the image editing processing unit 125 on the left-eye display 107 and the right-eye display 108. For example, the image is gamma-corrected and then displayed on the left-eye display 107 and the right-eye display 108.

[0048] Description of the line-of-sight detection operation

[0049] Figure 2 is a diagram explaining the principle of the line-of-sight detection method and shows the optical system for performing the processing of the aforementioned left-eye line-of-sight detection unit 116 and right-eye line-of-sight detection unit 117. In Figure 2 , the light sources 1006a and 1006b are light sources such as light-emitting diodes that emit infrared light that is not perceptible to the user. These light sources are substantially symmetric with respect to the optical axis of the light-receiving lens 1005 and irradiate the user's eyeball 1001. A part of the illumination light reflected by the eyeball 1001 is converged on the eyeball image sensor 1004 by the light-receiving lens 1005.

[0050] Figure 3 The reference numeral 3a in Figure 3 represents a schematic diagram of the eyeball image projected onto the eyeball image sensor 1004, and Figure 4 The reference numeral 3b in

[0051] When starting the line-of-sight detection routine in Figure 4 , in step S1201, the control unit 111 turns on the light sources 1006a and 1006b and irradiates the observer's eyeball 1001 with infrared light. The observer's eyeball image irradiated by the infrared light is formed on the eyeball image sensor 1004 by the light-receiving lens 1005.

[0052] In step S1202, the control unit 111 causes the eyeball image sensor 1004 to perform photoelectric conversion on the formed eyeball image and obtains an image signal of the eyeball.

[0053] In step S1203, the control unit 111 obtains from the eyeball image signal obtained in step S1202 the coordinates of the points corresponding to the corneal reflection images Pd and Pe and the pupil center c of the light sources 1006a and 1006b shown in Figure 2 .

[0054] Infrared light emitted from light sources 1006a and 1006b irradiates the cornea 1003 of the observer's eyeball 1001. At this time, corneal reflection images Pd and Pe formed by a part of the infrared light reflected from the surface of the cornea 1003 are converged by the light receiving lens 1005 and formed on the eyeball image sensor 1004 (points Pd' and Pe' in the figure). Similarly, light beams from ends a and b of the pupil 1002 are also imaged on the eyeball image sensor 1004.

[0055] Figure 3 Reference numeral 3a in the figure indicates an example image of the reflection image obtained from the eyeball image sensor 1004. Figure 3 Reference numeral 3b in the figure indicates the example luminance information related to Figure 3 region α in the figure.

[0056] As Figure 3 shown, the X-axis represents the horizontal direction, and the Y-axis represents the vertical direction. Here, Xd and Xe respectively represent the coordinates of images Pd' and Pe' formed by the light beams of the corneal reflection images from light sources 1006a and 1006b in the X-axis direction (horizontal direction). Xa and Xb respectively represent the coordinates of images a' and b' formed by the light beams from ends a and b of the pupil 1002 in the X-axis direction.

[0057] In Figure 3 the example luminance information represented by 3b in the figure, extremely high luminance levels are obtained at positions Xd and Xe corresponding to images Pd' and Pe' formed by the light beams of the corneal reflection images from light sources 1006a and 1006b. Except for the positions of Xd and Xe mentioned above, extremely low luminance levels are obtained in the region corresponding to the area of the pupil 1002 between coordinates Xa and Xb. At the same time, intermediate values between the above two luminance levels are obtained in the regions corresponding to the area of the iris 1101 outside the pupil 1002, where the X coordinate value is less than Xa and the X coordinate value is greater than Xb.

[0058] Based on the information related to the change in luminance level with respect to the X - coordinate position, the X - coordinates Xd and Xe of the images Pd’ and Pe’ formed by the light beams of the corneal reflection images from light sources 1006a and 1006b, and the X - coordinates Xa and Xb of the images a’ and b’ at the pupil ends can be obtained. In addition, when the rotation angle θx of the optical axis of the eyeball 1001 with respect to the optical axis of the light - receiving lens 1005 is small, the coordinate Xc of the light spot (denoted as c’) corresponding to the pupil center c whose image is formed on the eyeball image sensor 1004 can be expressed as Xc≈(Xa + Xb) / 2. Thus, the X - coordinate of c’ corresponding to the pupil center whose image is formed on the eyeball image sensor 1004 and the coordinates of the images Pd’ and Pe’ corresponding to the corneal reflection images of the light sources 1006a and 1006b can be estimated.

[0059] Return to Figure 4 Referring to the description of Figure 4 , in step S1204, the control unit 111 calculates the imaging magnification β of the eyeball image. β represents the magnification determined by the position of the eyeball 1001 with respect to the light - receiving lens 1005, and can be substantially obtained as a function of the distance (Xd - Xe) between the corneal reflection images Pd’ and Pe’.

[0060] In step S1205, the control unit 111 calculates the rotation angles θx and θy of the optical axis of the eyeball 1001 in two axial directions. The X - coordinate of the mid - point between the corneal reflection images Pd and Pe and the X - coordinate of the curvature center O of the cornea 1003 are substantially the same. Therefore, when Oc represents the standard distance from the curvature center O of the cornea 1003 to the center c of the pupil 1002, the rotation angle θx of the optical axis of the eyeball 1001 in the Z - X plane can be obtained from the following relational expression:

[0061] β*Oc*sinθx≈{(Xd + Xe) / 2}-Xc ···(1)

[0062] Although Figure 2 and Figure 3 show an example of calculating the rotation angle θx when the observer's eyeball rotates in a plane perpendicular to the Y - axis, the same method applies to calculating the rotation angle θy when the observer's eyeball rotates in a plane perpendicular to the X - axis.

[0063] In step S1206, the control unit 111 uses the θx and θy calculated in step S1205 to determine the position of the observer's line of sight on the left - eye display 107 and the right - eye display 108 (the position of the point where the observer is gazing: hereinafter referred to as the “gaze point”). Assuming that the coordinates of the position of the gaze point are (Hx, Hy) corresponding to the center c of the pupil 1002 on the left - eye display 107 and the right - eye display 108, Hx and Hy can be calculated as follows:

[0064] Hx = m × (Ax × θx + Bx) (2)

[0065] Hy = m × (Ay × θy + By) (3)

[0066] Here, the coefficient m is a constant determined by the configuration of the optical system, and is a conversion coefficient for converting the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 1002 on the left-eye display 107 and the right-eye display 108. In addition, it is assumed that the value of the coefficient m is predetermined and stored in the memory unit 113.

[0067] Ax, Bx, Ay, and By are line-of-sight correction coefficients for correcting individual differences in the observer's line of sight, and are obtained by performing a calibration operation. It is assumed that these coefficients are stored in the memory unit 113 before the start of the line-of-sight detection routine.

[0068] After calculating the coordinates (Hx, Hy) of the center c of the pupil 1002 on the left-eye display 107 and the right-eye display 108 as described above, the control unit 111 stores the above coordinates in the memory unit 113 in step S1207, and ends the line-of-sight detection routine.

[0069] The above method for obtaining the fixation point coordinates on the left-eye display 107 and the right-eye display 108 uses the corneal reflection images of the light sources 1006a and 1006b. However, the line-of-sight detection method is not limited to the above method, and any other method can be used as long as the eye rotation angle can be obtained from the captured eye image.

[0070] Overlooking degree calculation and blur area change

[0071] will be described with reference to Figures 5A to 5C , Figure 6 , Figure 7A and Figure 7B to describe the calculation of the overlooking degree and the change of the blur area. Figures 5A to 5C is a diagram showing an example of a method for detecting the convergence angle and a method for calculating the overlooking degree. Figure 6 is a flowchart showing an example of changing the blur area based on the overlooking degree. Figure 7A and Figure 7B show an example of a process for changing the area (blur area) of the rendering process of the video based on the overlooking degree determination result.

[0072] In Figure 6 step S301, the control unit 111 calculates the convergence angle of the eyes. The "convergence angle" refers to as Figure 5AThe angle θ1 shown in , formed by the lines of sight of the two eyes when viewing point P0. The convergence angle θ1 decreases as the distance to point P0 increases, and conversely, the convergence angle θ1 increases as the distance to point P0 decreases. The convergence angle can be calculated by obtaining the intersection point of the lines in the directions of the fixation point positions of the respective eyes.

[0073] In this embodiment, for example, Figure 1B the left-eye line-of-sight detection unit 116 and the right-eye line-of-sight detection unit 117 in calculate Figure 5B and Figure 5C the line-of-sight vectors 200 and 201 of the respective eyes in . The line-of-sight vectors 200 and 201 are input to the object fixation determination unit 123. The object fixation determination unit 123 obtains the convergence angle from the line-of-sight vectors of the respective eyes.

[0074] In step S302, the control unit 111 calculates the distance D1 to the fixation position (first distance detection) using the convergence angle obtained in step S301 and the distance between the two eyes (from point OL to point OR). This can be obtained using trigonometric functions. Alternatively, the following method can be applied: The correlation between a plurality of subjects at different distances and the convergence angles when viewing these subjects is measured and stored in advance, and the distance to the subject is estimated based on this correlation according to the convergence angle.

[0075] In step S303, the control unit 111 calculates the subject distance D2 (second distance detection) based on the information obtained from the left-eye camera 103 and the right-eye camera 104. For example, the depth information obtained from the left-eye camera 103 and the right-eye camera 104 is used to perform the calculation for Figure 5B and Figure 5C the subject distance D2 of the subject existing on the line 203 between the line-of-sight vectors 200 and 201 of the respective eyes in . More specifically, the autofocus function (phase difference detection function) of the left-eye camera 103 and the right-eye camera 104 is used to calculate the focusing position of the focusing lens, and the subject distance is calculated based on the position of the focused focusing lens and the parameters of the optical system. However, the method of calculating the subject distance D2 is not limited to this method, and if the distance from the HMD 100 to the subject can be obtained, any method such as a distance calculation method using LiDAR can be used.

[0076] In step S304, the control unit 111 calculates a gaze degree for determining whether the user is gazing at the subject. The ratio between the gaze distance D1 obtained in step S302 and the subject distance D2 obtained in step S303 is obtained, and the gaze degree is calculated by obtaining the degree of deviation between the gaze distance and the subject distance. The gaze degree is, for example, calculated as the ratio (D1 / D2) between the gaze distance D1 and the subject distance D2. However, the method of calculating the gaze degree is not limited to this, and the gaze degree can alternatively be represented using the difference between the gaze distance D1 and the subject distance D2.

[0077] Based on the calculated gaze degree, the object gaze determination unit 123 determines that the user is gazing at the subject less as the ratio between the gaze distance D1 and the subject distance D2 further deviates from 1, and determines that the user is gazing at the subject more as the ratio approaches 1. Note that if the gaze degree (D1 / D2) is greater than 1, the user is viewing something farther away from the subject, and the convergence angle θ1 takes a small value. If the gaze degree (D1 / D2) is less than 1, the user is viewing something closer than the subject, and the convergence angle θ1 takes a large value. In either case, it is determined that the user is not gazing at the subject.

[0078] In step S305, the control unit 111 uses the bird's-eye view degree determination unit 124 to calculate a bird's-eye view degree for indicating whether the user is viewing the subject in a bird's-eye view manner based on the gaze degree of the subject obtained in step S304. As the gaze degree obtained in step S304 deviates from 1, it is determined that the bird's-eye view degree is higher, and as the bird's-eye view degree approaches 1, it is determined that the bird's-eye view degree is lower. Specifically, the bird's-eye view degree is obtained, for example, by the formula: bird's-eye view degree = gaze degree (Dl / D2) - 1, and as the absolute value of the value obtained by this formula is larger, the bird's-eye view degree is determined to be higher. Further, if the bird's-eye view degree (Dl / D2) - 1 takes a positive value, it indicates that the user is viewing something farther away from the subject, and if the bird's-eye view degree takes a negative value, it indicates that the user is viewing something closer than the subject. However, the method of calculating the bird's-eye view degree is not limited to this, and the bird's-eye view degree can alternatively be represented using, for example, the reciprocal of the gaze degree (Dl / D2).

[0079] In step S306, the blurred area in the image rendering is changed based on the bird's-eye view degree obtained in step S305. For example, Figure 7A is a diagram showing the setting of the blurred area when it is determined that the bird's-eye view degree is lower than a predetermined threshold. When the bird's-eye view degree is low, the convergence angle formed by the eye's sight vectors 200 and 201 is large, such that the gaze positions of the respective eyes in the left sight camera unit 105 and the right sight camera unit 106 are close to each other. That is, the user is viewing the area near the gaze position in the image 400, and the image quality at this part needs to be high. Therefore, as Figure 7AAs shown, the blurred area when the overlooking degree is low is, for example, the area outside circles 403a and 404a in a fixed area centered on the fixation position of each eye (i.e., the peripheral area of the image). Note that the images within circles 403a and 404a are not edited and maintain high image quality.

[0080] Figure 7B FIG. is a diagram showing the setting of the blurred area when it is determined that the overlooking degree is greater than or equal to a predetermined threshold. When it is determined that the overlooking degree is high, especially when the user is viewing something farther from the subject, the convergence angle formed by the line-of-sight vectors 200 and 201 of the eyes is small, causing the fixation positions of the respective eyes in the left line-of-sight imaging unit 105 and the right line-of-sight imaging unit 106 to be separated from each other. That is, the user is viewing a relatively wide area near the fixation position in the image 400, and the image quality at this part needs to be high. Therefore, as Figure 7B shown, the blurred area when the overlooking degree is high is, for example, the area outside circles 403b and 404b centered on the fixation position of each eye (i.e., the peripheral area of the image).

[0081] Note that in the above description, the blurred area is the area outside the circle centered on the fixation position. However, the blurred area is not limited to this, and for example, it can be the area outside a rectangular area including the fixation position. In addition, in the above description, the size of the blurred area is gradually changed according to the overlooking degree. However, the size of the blurred area can be continuously changed alternatively according to the change in the overlooking degree.

[0082] Therefore, it is determined whether the user is viewing the video in an overlooking manner, and the rendering area is changed according to the determined overlooking degree. Thus, if the overlooking degree is high (the fixation degree is low), rendering can be performed while setting the area size of the blurred area in the image to a relatively narrow area in the peripheral area of the image (i.e., setting a relatively wide area including the fixation position as the high-image-quality area). If the overlooking degree is low (the fixation degree is high), rendering can be performed while setting the area size of the blurred area in the image to a relatively wide area in the peripheral area of the image (i.e., setting a relatively narrow area including the fixation position as the high-image-quality area). This makes it possible to provide a viewing environment in which the appropriate area of the high-image-quality area is set according to the user's overlooking degree.

[0083] Second Embodiment

[0084] The second embodiment of the present invention will be described below. The image processing device in the second embodiment has the same configuration as the Figure 1A and 1B shown head-mounted display (hereinafter referred to as "HMD") 100, and thus its description is omitted.

[0085] In this embodiment, an example of changing the rendering image quality according to the overlooking degree will be described. Figure 8 is a flowchart showing an example of changing the rendering image quality according to the overlooking degree. Since Figure 8 the processing in Figure 6 has many common parts with that of the first embodiment illustrated, the steps of performing the same processing as in Figure 6 are assigned the same step numbers as in Figure 6 and their descriptions are omitted.

[0086] In Figure 8 the processing in steps S301 to S305 is the same as the processing in steps S301 to S305 in Figure 6 .

[0087] In step S506, the control unit 111 changes the image quality in image rendering based on the overlooking degree obtained in step S305.

[0088] Figure 9A and Figure 9B show examples of changing the rendering image quality based on the overlooking degree.

[0089] For example, if it is determined that the overlooking degree is lower than a predetermined threshold, that is, if the user is gazing at the subject as shown in Figure 9A , rendering processing is performed such that the image quality in the rendering area 603 in the image 600 is high, that is, higher than the predetermined image quality. Rendering processing is performed on other areas with a low image quality (that is, an image quality lower than the predetermined image quality).

[0090] The image quality described here refers to, for example, the resolution. In the high image quality area, rendering is performed to obtain a detailed image with a predetermined level of resolution, and in the low image quality area, rendering processing is performed at a resolution lower than the predetermined level.

[0091] As shown in Figure 9B , if the overlooking degree is greater than or equal to the predetermined threshold, that is, if the user is not gazing at a specific subject but is viewing the scene in a somewhat overlooking manner, rendering processing is performed while setting the image quality in the rendering area 603 to a medium image quality. Rendering processing is performed on other areas with a low image quality which is an image quality lower than the predetermined image quality. Here, the "medium image quality" refers to a resolution that is higher than the image quality outside the rendering area and lower than the case where no rendering (that is, no image editing) is performed.

[0092] Note that in Figure 9B the rendering area when the overlooking degree is greater than or equal to the predetermined threshold is depicted as the same rendering area as in the case where the overlooking degree is lower than the predetermined level shown in Figure 9A .

[0093] As described above, in the present embodiment, it is determined whether the user is watching the video in an overlooking manner. If the overlooking degree is high, the resolution as the rendering image quality is set low, and if the overlooking degree is low, the resolution is set high. This enables providing an appropriate image quality to the user according to the overlooking degree.

[0094] Third Embodiment

[0095] A third embodiment of the present invention will be described below. The image processing apparatus of the third embodiment has the same configuration as the head-mounted display (hereinafter referred to as "HMD") 100 shown in the first embodiment, and thus its description is omitted. Figure 1A and Figure 1B shown, and thus its description is omitted.

[0096] In this embodiment, an example of changing the blurring area and the rendering image quality according to the overlooking degree will be described. Figure 10 is a flowchart showing an example of changing the blurring area and the rendering image quality according to the overlooking degree. Since Figure 10 the processing in is similar to that of the flowchart exemplifying the first embodiment Figure 6 and has many common parts, the steps of performing the same processing as in Figure 6 are assigned the same step numbers as in Figure 6 and their description is omitted.

[0097] In Figure 10 the processing in steps S301 to S305 is the same as the processing in steps S301 to S305 in Figure 6 .

[0098] In step S706, as in step S306 in Figure 6 , the control unit 111 changes the blurring area in image rendering based on the overlooking degree obtained in step S305. For example, Figure 11A is a diagram showing the setting of the blurring area when it is determined that the overlooking degree is lower than a predetermined threshold. When the overlooking degree is low, the convergence angle formed by the line-of-sight vectors 200 and 201 of the eyes is large, such that the fixation positions of the respective eyes in the left line-of-sight imaging unit 105 and the right line-of-sight imaging unit 106 are close to each other. That is, the user is watching the area near the fixation position in the image 900, and the image quality at this part needs to be high. Therefore, as shown in Figure 11A , the blurring area when the overlooking degree is low is, for example, the area outside the circles 903a and 904a in a fixed area centered on the fixation position of each eye.

[0099] Figure 11BThis is a diagram showing the setting of the blurred area when it is determined that the overlooking degree is greater than or equal to a predetermined threshold. When it is determined that the overlooking degree is high, especially when the user is viewing something farther away from the subject, the convergence angle formed by the line-of-sight vectors 200 and 201 of the eyes is small, so that the fixation positions of the eyes in the left line-of-sight imaging unit 105 and the right line-of-sight imaging unit 106 are separated from each other. That is, the user is viewing a relatively wide area near the fixation position in the image 900, and the image quality at this part needs to be high. Therefore, as Figure 11B shown, the blurred area when the overlooking degree is high is, for example, the area outside the circles 903b and 904b centered on the fixation positions of each eye. Note that in the above description, the blurred area is the area outside the circle centered on the fixation position. However, the blurred area is not limited to this, and for example, it can be the area outside the rectangular area including the fixation position.

[0100] In step S707, as in step S506 in Figure 8 , the control unit 111 changes the image quality in image rendering based on the overlooking degree obtained in step S305.

[0101] For example, if it is determined that the overlooking degree is lower than the predetermined threshold, that is, if the user is fixating on the subject as Figure 11A shown, rendering processing is performed so that the image quality in the rendering areas 903a and 903b in the image 900 is high, that is, higher than the predetermined image quality. Rendering processing is performed on other areas with low image quality (that is, lower than the predetermined image quality).

[0102] The image quality described here refers to, for example, the resolution. In the high image quality area, rendering is performed to obtain a detailed image with a resolution higher than a predetermined level, and in the low image quality area, rendering processing is performed at a resolution lower than the predetermined level.

[0103] As Figure 11B shown, if the overlooking degree is greater than or equal to the predetermined threshold, that is, if the user is not fixating on a specific subject but is viewing the scene in a somewhat overlooking manner, rendering processing is performed while setting the image quality in the rendering areas 903b and 904b to medium image quality. Rendering processing is performed on other areas with low image quality, which is lower than the predetermined image quality. Here, the "medium image quality" refers to a resolution that is higher than the image quality outside the rendering area and lower than the case where no rendering (i.e., no image editing) is performed.

[0104] As described above, in the present embodiment, it is determined whether the user is viewing the video in an aerial view manner. If the aerial view degree is high, the area size of the blurred area in the peripheral area of the image is set to be narrow, and the resolution as the rendering image quality (the resolution near the fixation point) is set to be low. If the aerial view degree is low, the area size of the blurred area in the peripheral area of the image is set to be wide, and the resolution as the rendering image quality (the resolution near the fixation point) is set to be high. This makes it possible to provide an appropriate blurred area and appropriate image quality to the user according to the aerial view degree.

[0105] Note that in the above description, the size of the blurred area and the resolution as the rendering image quality gradually change according to the aerial view degree. However, the size of the blurred area and the resolution as the rendering image quality may also change continuously according to the change in the aerial view degree.

[0106] In Figure 10 only the blurred area obtained in step S706 and the rendering image quality obtained in step S707 are set based on the user's aerial view degree. However, the blurred area obtained in step S706 and the rendering image quality obtained in step S707 may also change based on the movement of the subject.

[0107] Figure 12 is a flowchart showing an example of changing the blurred area and the rendering image quality while considering the movement of the subject.

[0108] In Figure 12 the processing in steps S301 to S305 is also the same as the processing in steps S301 to S305 in Figure 6 The processing in steps S301 to S305 in

[0109] In step S806, the control unit 111 measures the movement of the subject. For example, on each display, the movement speed of the subject is measured based on the moving distance on the image plane of the subject and the movement of the HMD 100.

[0110] In step S807, the control unit 111 sets the blurred area based on the aerial view degree obtained in step S305 and the movement of the subject obtained in step S806. For example, even if the aerial view degree is lower than or equal to a predetermined level, if the movement speed of the subject is higher than the predetermined level, the radius of the high-image-quality rendering area centered on the line-of-sight vector of each eye is also expanded. In addition, even if the aerial view degree is higher than the predetermined level, if the movement speed of the subject is lower than or equal to the predetermined level, the radius of the rendering area centered on the line-of-sight vector of each eye is also set to be narrow.

[0111] In step S808, the control unit 111 sets the resolution based on the movement of the subject obtained in step S806. For example, even if the user's bird's-eye view degree is higher than a predetermined level, if the movement speed of the subject is higher than a predetermined level, the resolution in the rendering areas 903b and 904b in Figure 11B is set to a medium resolution. If the movement speed of the subject is lower than or equal to the predetermined level, the resolution in the rendering areas 903b and 904b is set to a high resolution.

[0112] In step S809, the control unit 111 sets the frame rate based on the movement of the subject obtained in step S806. Even when the user's bird's-eye view degree is high, if the movement speed of the subject is higher than a predetermined level, the frame rate is also set to be higher than a predetermined level, and if the movement speed of the subject is lower than or equal to the predetermined level, the frame rate is also set to be lower than or equal to the predetermined level. The rendering process is performed at the frame rate set in this way.

[0113] In step S810, the bit rate is set based on the movement of the subject obtained in step S806. Even when the user's bird's-eye view degree is higher than a predetermined level, if the movement speed of the subject is higher than a predetermined level, the bit rate is also set to be higher than a predetermined level, and if the movement speed of the subject is lower than or equal to the predetermined level, the bit rate is also set to be lower than or equal to the predetermined level. The rendering process is performed at the bit rate set in this way.

[0114] If the bird's-eye view degree is high, the details of the subject cannot be recognized, so the resolution is set to be lower than the predetermined rendering resolution. On the other hand, if there is a fast-moving subject, a video with a sense of visual presence is displayed to present a smoother and more natural movement.

[0115] Even if the bird's-eye view degree is high, it is not necessary to present a smooth movement in a scene with less movement. Instead, the resolution is increased to provide a fine video.

[0116] As described above, in Figure 12 , the blurring area and the rendering image quality are changed based on the movement of the subject. However, the rendering area and the rendering image quality can be changed not only based on the movement of the subject, but also based on the position, brightness, or other aspects of the shooting scene.

[0117] For the rendering image quality, the resolution, the frame rate, and the bit rate are changed, but the items to be changed are not limited to these. For example, in a dark place, the dynamic range can also be widened to widen the band of brightness information and improve the reproducibility of light and shade and color tone.

[0118] Other embodiments

[0119] The present invention can also be implemented by the following processes: A program that implements one or more functions of the above-described embodiments is provided to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus load and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0120] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, the appended claims are made to increase the disclosure of the scope of the present invention.

[0121] This application claims priority based on Japanese Patent Application No. 2022-195015 filed on December 6, 2022, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus for processing an image to be displayed on a display unit, the image processing apparatus comprising: a line-of-sight detection unit configured to detect a direction of a user's line of sight viewing the display unit; a first distance detection unit configured to detect a first distance, which is a distance to a position at which the user is gazing, based on the detected direction of the line of sight; a second distance detection unit configured to detect a second distance, which is a distance to a subject; an acquisition unit configured to acquire an overlooking degree indicating a degree to which the user is viewing the subject in an overlooking manner based on the first distance and the second distance; and a change unit configured to change at least one of the following based on the overlooking degree: a region in which a process for changing an image quality of the image is performed and the image quality in the process for changing the image quality.

2. The image processing apparatus according to claim 1, wherein The process for changing the image quality is a process for blurring a peripheral region of the image.

3. The image processing apparatus according to claim 2, wherein As the overlooking degree increases, the change unit reduces a region size of the peripheral region of the image on which the blurring process is performed.

4. The image processing apparatus according to claim 1, wherein The process for changing the image quality is a process for making an image quality of an image near the user's viewpoint higher than an image quality in the peripheral region of the image.

5. The image processing apparatus according to claim 4, wherein As the overlooking degree increases, the change unit reduces the image quality in a region where the image quality of the image increases.

6. The image processing apparatus according to claim 4 or 5, wherein The image quality of the image is a resolution of the image.

7. The image processing apparatus according to claim 1, wherein The process for changing the image quality is a process for blurring a peripheral region of the image and increasing an image quality of an image near the user's viewpoint.

8. The image processing apparatus according to claim 7, wherein As the overlooking degree increases, the change unit reduces a region size of the peripheral region of the image on which a process for blurring the image is to be performed, and reduces the image quality in a region where the image quality of the image increases.

9. The image processing apparatus according to claim 1, wherein The change unit changes a region in which a process for changing the image quality is performed based on a shooting scene.

10. The image processing apparatus according to claim 1, wherein The change unit changes the image quality in the process for changing the image quality based on the shooting scene.

11. The image processing apparatus according to claim 10, wherein The change unit changes a bit rate, a frame rate, and a dynamic range of the image based on the shooting scene.

12. The image processing apparatus according to any one of claims 1 to 11, characterized in that, The acquisition unit acquires a gazing degree indicating a degree to which the user is gazing at the subject based on a ratio between the first distance and the second distance.

13. The image processing apparatus according to claim 12, wherein The acquisition unit acquires the overlooking degree based on the gazing degree.

14. An image processing method for processing an image to be displayed on a display unit, the image processing method comprising: a line-of-sight detection step of detecting a direction of a user's line of sight viewing the display unit; a first distance detection step of detecting a first distance, which is a distance to a position at which the user is gazing, based on the detected direction of the line of sight; a second distance detection step of detecting a second distance, which is a distance to a subject; an acquisition step of acquiring an overlooking degree indicating a degree to which the user is viewing the subject in an overlooking manner based on the first distance and the second distance; and A changing step for changing at least one of the following based on the overlooking degree: a region for performing a process for changing the image quality of the image and the image quality in the process for changing the image quality of the image.

15. A program for causing a computer to execute each step of the image processing method according to claim 14.

16. A computer-readable storage medium having stored thereon a program for causing a computer to execute each step of the image processing method according to claim 14.

Images