Information processing device and method, and program

Abstract

This invention provides an information processing device that reduces power consumption without compromising the user's viewing experience by changing the quality of the displayed image according to the user's usage. [Solution] The information processing device 110 has a control unit 211 and a content DB 212. The control unit 211 is an information processing device for displaying a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, and includes a position acquisition unit 301 that acquires the position of the visual organs, a change amount acquisition unit 302 that acquires the change amount between the first position and the second position of the visual organs acquired by the position acquisition unit 301, and an image quality determination unit 303 that determines the quality of the display image according to the change amount acquired by the change amount acquisition unit 302.

Description

[Technical Field]

[0001] This disclosure relates to information processing equipment and methods, as well as programs. [Background technology]

[0002] Traditionally, in cross-reality (XR) systems that allow users to experience virtual reality, head-mounted displays (HMDs) have been used to display the virtual space. An HMD is a glasses-type device equipped with a small display, and it is envisioned that users will wear it on their heads for precise work verification, or hold it in their hands for easy video experiences without wearing it on their heads. Furthermore, it is envisioned that it will be used not only while staying in one place, but also while moving. When using an HMD while the head is moving, there is a possibility that the positional relationship between the HMD and the user's head may shift.

[0003] Patent Document 1 discloses a method in which the position of visual organs such as the inner and outer corners of the eyes is measured by the HMD's camera, the difference from the standard position is detected as a positional deviation, and image processing is performed so that the 3D image displayed on the display rotates or translates according to the detected positional deviation. Patent Document 2 discloses a method for adjusting the display range of the display to match the observer's field of view, depending on the observer's wearing posture. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Patent No. 5414946 [Patent Document 2] Japanese Patent Publication No. 2010-232718 [Overview of the project] [Problems that the invention aims to solve]

[0005] When the movement of the HMD (Head-Mounted Display) matches the movement of the user's face, such as in head-mounted styles, high image quality is required. On the other hand, when the movement of the HMD does not match the movement of the user's face, such as when the user holds the HMD in one hand while walking around, the image quality does not need to be as high. Generally, high-quality image processing increases power consumption. In standalone XR experiences where the user carries the entire system, including the battery and other power sources, providing consistently high-quality images would shorten the experience duration.

[0006] However, the technology disclosed in Patent Document 1 always performs high-quality display image processing even when the movement of the HMD and the movement of the user's face do not match, so power consumption remains high. Also, the technology disclosed in Patent Document 2 adjusts the display range of the display according to the user's field of view, but does not adjust the quality of the display image, so power consumption remains almost unchanged.

[0007] This disclosure has been made in view of the above-mentioned issues, and aims to provide an information processing device that can reduce power consumption without lowering the value of the user's video experience by determining the quality of the displayed image according to the user's usage conditions. [Means for solving the problem]

[0008] The information processing device disclosed herein is an information processing device for displaying a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of a visual organ, and comprises a position acquisition unit, a change amount acquisition unit, and an image quality determination unit. The position acquisition unit acquires the position of the visual organ. The change amount acquisition unit acquires the change amount between a first position and a second position of the visual organ acquired by the position acquisition unit. The image quality determination unit determines the quality of the display image according to the change amount acquired by the change amount acquisition unit. [Effects of the Invention]

[0009] According to the present disclosure, an information processing apparatus is realized that can reduce power consumption without degrading the video experience value of the user by determining the quality of the display image according to the usage situation of the user.

Brief Description of the Drawings

[0010] [Figure 1] FIG. 8 is a perspective view showing the appearance of the HMD system according to the present embodiment. [Figure 2] FIG. 11 is a front view showing the appearance of the HMD main body according to the present embodiment. [Figure 3] FIG. 14 is a block diagram showing the internal configuration of the HMD system according to the present embodiment. [Figure 4] FIG. 17 is a schematic diagram showing the structure of the human eye. [Figure 5] FIG. 20 is a block diagram showing the main functions of the image processing apparatus of the HMD system according to the present embodiment. [Figure 6] FIG. 23 is a flowchart showing the image processing according to the present embodiment. [Figure 7] FIG. 26 is a state transition diagram showing a method for determining the quality of the display image.

Mode for Carrying Out the Invention

[0011] -Specific Description of the Present Embodiment- Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential, and the plurality of features may be arbitrarily combined. Further, in the drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted. The details of the dimensions and structures shown in the embodiments are not limited to those shown in the text and the drawings.

[0012] [External Configuration of the HMD System] FIG. 1 is a perspective view showing the appearance of the HMD system according to the present embodiment. The HMD system of this embodiment displays, as a display image, a composite image of a virtual space image or a real space image and a virtual space image in the vicinity of the user's face.

[0013] In this HMD system, the HMD main body 100 is connected from a mounting portion 101 that is worn around the user's head in a headband shape via a connecting portion 102 in order to fix the position in front of the user's eyes. By removing the connecting portion 102, it is also possible to use only the HMD main body 100 by hand. A plurality of imaging units 202 for photographing external images are arranged on the HMD main body 100.

[0014] On the HMD main body 100, a composite image in which an imaging image obtained by the HMD main body 100 imaging the front range of the user and content such as CG (computer graphics) in a form corresponding to the posture of the HMD main body 100 are combined is displayed.

[0015] The image processing device 110 is a control device (electronic device) that controls the HMD main body 100. The image processing device 110 is, for example, a smartphone, a tablet terminal, or a PC (personal computer). The image processing device 110 is connected to the HMD main body 100 wirelessly or by wire. The image processing device 110 generates a composite image by combining the imaging image and the CG, and transmits the composite image to the HMD main body 100. Note that each component of the image processing device 110 may be provided inside the HMD main body 100.

[0016] FIG. 2 is a front view showing the appearance of the HMD main body 100 according to this embodiment. (a) shows the appearance as seen from the user side wearing the HMD main body 100, and (b) shows the appearance around the right display unit 104. In FIG. 2(a), the right display unit 104 includes an image display unit 203 described later and an eyepiece lens (not shown) that is a display optical system, and can be driven in the horizontal direction to change its position. On the other hand, if the center line of the mounting portion 101 is defined by a broken line in FIG. 2, the left display unit 105 is configured symmetrically with the right display unit 104 with respect to this center line, and can similarly change its position in the horizontal direction.

[0017] In Figure 2(b), the right-side display unit 104 includes an eyepiece optical system, an image display unit 203, and a visual organ detection unit 204. In this embodiment, "visual organs" (eye organs) mainly consist of the eyeball and optic nerve, and include accessories such as eyelids, extraocular muscles, and lacrimal apparatus. In other words, it includes not only the eyeball but also all organs around the eye (eyelids, inner corner of the eye, outer corner of the eye, lacrimal caruncle, inner canthus, outer canthus, iris, pupil, etc.).

[0018] Furthermore, the right-side display unit 104 can be driven left and right by a stepping motor 205. The left-side display unit 105 is configured similarly, and the positions of the right-side display unit 104 and the left-side display unit 105 can be controlled independently. The position of the user's visual organs is detected from the image acquired by the visual organ detection unit 204, the target driving positions of the display units 104 and 105 are determined, and the stepping motor 205 is driven. This performs what is known as IPD (Interpupillary Distance) adjustment, which adjusts the display unit to the optimal position according to the user's interpupillary distance. In addition to automatic IPD adjustment by driving with the stepping motor 205, the user can also perform manual IPD adjustment by moving the display unit themselves.

[0019] [Internal Configuration of HMD] Figure 3 is a block diagram showing the internal configuration of the HMD system according to this embodiment. The HMD control unit 201 includes a CPU (microcomputer) as the central processing unit and comprehensively controls each part of the HMD body 100, performing various settings. When the HMD control unit 201 acquires a composite image (an image created by combining an image captured by the imaging unit 202 of the space in front of the user with computer graphics) from the image processing unit 110, it displays the composite image on the image display unit 203. As a result, the user can view the composite image displayed on the image display unit 203 by wearing the HMD body 100. The user can then experience various mixed realities, such as computer graphics merging with real space.

[0020] The imaging unit 202 includes two cameras (imaging devices). The two cameras are positioned near the user's left and right eyes when the HMD body 100 is worn, in order to capture the same space as the space the user sees without wearing the HMD. The images (captured images) captured by the two cameras of the subject (the area in front of the user) are output to the image processing device 110. In addition, the two cameras in the imaging unit 202 can acquire distance information from the two cameras to the subject by measuring distance using stereo cameras. Furthermore, by placing another camera and analyzing the images from these cameras in a time series, it is possible to obtain information on the position and orientation of the HMD in real space.

[0021] The image display unit 203 displays a composite image. The image display unit 203 has, for example, a liquid crystal panel or an organic EL panel. When the user is wearing the HMD main unit 100, images corresponding to the user's left and right eyes are displayed separately. It is also possible to use a device with a semi-transparent half-mirror for the image display unit 203. In this case, for example, the image display unit 203 may display an image so that the CG is superimposed directly onto the real space visible through the half-mirror, using a technology commonly known as AR. Alternatively, the image display unit 203 may display an image of a completely virtual space without using captured images, using a technology commonly known as VR.

[0022] The visual organ detection unit 204 is a camera that captures images of the area around the user's eyes. The captured images are output to the image processing device 110. Infrared light may be used to capture infrared images when photographing the area around the user's eyes.

[0023] Figure 4 is a schematic diagram showing the structure of the human eye. When the user wears the HMD on their head or holds it close to their eyes, as shown in Figure 4, parts of the eyeball, such as the lacrimal caruncle, medial canthus, lateral canthus, iris, and pupil, which are part of the visual organs (eye organs), are captured in the image. The positions of these parts within the captured image are identified by image processing. In this embodiment, we focus on the position of the lacrimal caruncle as a specific example of a visual organ and explain the example by identifying its coordinates, but as shown in Figure 4, there are other visual organs as well, and this is not the only example.

[0024] The IPD drive unit 205 is a stepping motor and can drive the display unit according to instructions from the image processing device 110.

[0025] The attitude sensor unit 206 detects the attitude and position of the HMD body 100. The attitude sensor unit 206 includes, for example, an inertial measurement unit (IMU). The attitude sensor unit 206 outputs the attitude information of the HMD body 100 as attitude information to the image processing device 110.

[0026] The operation unit 207 is composed of, for example, buttons, and transmits the user's operation as input to the image processing device 110.

[0027] [Internal configuration of the image processing unit] The control unit 211 includes a CPU (microcomputer), which is a central processing unit, and comprehensively controls the image processing device 110 and performs various settings.

[0028] The control unit 211 receives the image (captured image) acquired by the imaging unit 202 of the HMD body 100 and the posture information acquired by the posture sensor unit 206 from the HMD control unit 201 of the HMD body 100. The control unit 211 then combines the captured image with an arbitrary CG to generate a composite image to be displayed on the image display unit 203 of the HMD body 100. The control unit 211 can also interpret instructions from the user using input from the operation unit 207. The control unit 211 transmits the composite image to the HMD control unit 201 of the HMD body 100.

[0029] The control unit 211 controls the position, orientation, and size of the CG in the composite image based on the information (distance information and orientation information) acquired by the HMD main unit 100. For example, when the control unit 211 places a virtual object represented by CG near a specific object that exists in the real world in the virtual space represented by the composite image, it increases the size of the virtual object (CG) as the distance between the specific object and the imaging unit 202 decreases. By controlling the position, orientation, and size of the CG in this way, the control unit 211 can generate a composite image in which CG objects that are not actually located in the real world appear as if they were located in the real world.

[0030] The content DB 212 is a memory unit that stores information such as computer graphics (CG). The control unit 211 can switch which CG is read from the content DB 212 (i.e., which CG is used to generate the composite image).

[0031] Figure 5 is a block diagram showing the main functions of the image processing device 110 of the HMD system according to this embodiment. The control unit 211 includes a position acquisition unit 301, a change amount acquisition unit 302, an image quality determination unit 303, an image quality modification unit 304, and a storage unit 305.

[0032] The position acquisition unit 301 acquires the position of the user's visual organs, such as the lacrimal caruncle, from the image captured by the visual organ detection unit 204.

[0033] The change amount acquisition unit 302 acquires the change amount between the first position and the second position of the user's visual organs, which are acquired by the position acquisition unit 301. The change amount acquisition unit 302 includes a relative distance acquisition unit 302a and a relative velocity acquisition unit 302b. The relative distance acquisition unit 302a acquires the relative distance between the first position, which is the initial position of the user's visual organs, and the second position, which is the moved position of the visual organs, as the change amount. Here, the second position is, for example, the position of the visual organs in the image detected from the captured image of a predetermined frame (frame N). The relative velocity acquisition unit 302b acquires the relative velocity between the first position and the second position, which has changed over time from the first position, as the change amount. Here, the first position is, for example, the position of the visual organs in the image detected from the captured image of a predetermined frame (frame N), and the second position is the position of the visual organs in the image detected from the captured image of the frame preceding frame N (frame N-1).

[0034] The image quality determination unit 303 determines the quality of the display image displayed on the image display unit 203 of the HMD body 100 according to the amount of change (relative distance and / or relative velocity) acquired by the amount of change acquisition unit 302. For example, if the amount of change is above a predetermined threshold for a first period while operating in the first image quality mode, the image quality determination unit 303 transitions to a second image quality mode, which has a lower display image quality than the first image quality mode. If the amount of change is below the threshold for a second period, which is shorter than the first period, while operating in the second image quality mode, the image quality determination unit 303 transitions to the first image quality mode.

[0035] The image quality modification unit 304 changes the quality of the display image shown in the image display unit 203 to the quality determined by the image quality determination unit 303. The image quality modification unit 304 changes, for example, at least one of the following: the resolution of the display image, brightness, frame rate, image processing calculation cycle, and the toggle on or off of image processing.

[0036] The memory unit 305 stores the computer program that drives the image processing device 110, and also stores the initial position (first position) of the user's visual organs acquired by the position acquisition unit 301.

[0037] [Image processing according to this embodiment] Figure 6 is a flowchart showing the image processing according to this embodiment. When the processing operation starts, IPD adjustment is performed in step S1001. IPD adjustment is performed when the user issues an automatic IPD start command by operating the control unit 207, and the control unit 211 drives the stepping motor 205 to perform automatic IPD adjustment. Alternatively, the user may perform manual IPD adjustment by moving the display unit themselves.

[0038] In step S1002, the position acquisition unit 301 of the control unit 211 identifies the position of the user's visual organs, such as the lacrimal caruncle, from the image captured by the visual organ detection unit 204.

[0039] In step S1003, the memory unit 305 stores the positions of the visual organs obtained in step S1002. For example, it stores the positions (Xss, Yss) of the visual organs in the captured image.

[0040] Steps 1001 through 1003 are processes performed only the first time, while steps 1004 onwards are performed when the HMD unit 100 has started normal operation and the user has begun their experience using the HMD unit 100.

[0041] In step S1004, similar to step S1002, the position acquisition unit 301 of the control unit 211 identifies the position of the user's visual organs from the image captured by the visual organ detection unit 204.

[0042] In step S1005, the change amount acquisition unit 302 acquires the amount of change between the position of the visual organ acquired in step S1004 and the initial position of the visual organ stored in step S1003. For example, the relative distance acquisition unit 302a of the change amount acquisition unit 302 calculates the relative distance d between the position (Xs1, Ys1) of the user's lacrimal caruncle detected from the captured image of a certain frame (frame N) and the stored initial position (Xss, Yss) of the lacrimal caruncle. The relative distance d is obtained by the following equation (1). d={(Xss-Xs1) 2 +(Yss-Ys1) 2} 1 / 2 ...(1) Here, d is the relative distance (in pixels), Xss and Yss are the in-image coordinates of the eye at the initial position (in pixels), and Xs1 and Ys1 are the current in-image coordinates of the eye (in pixels).

[0043] If the position of the visual organs cannot be determined, such as when the user's face is far from the HMD body 100 at step S1004, the relative distance acquisition unit 302a sets the following equation (2) and performs subsequent control. d = d_max ... (2) Here, d_max is set to a large value that cannot be calculated using normal operations.

[0044] In step S1006, the control unit 211 determines whether the relative distance d obtained in step S1005 is less than or equal to the threshold d_th. If it is determined that the relative distance d is less than or equal to the threshold d_th, the process proceeds to step S1007. If it is determined that the relative distance d is greater than the threshold d_th, the process proceeds to step S1009.

[0045] In step S1007, the control unit 211 clears the Lost_count, which is the number of times the position of the visual organs has been lost, to 0.

[0046] In step S1008, the change amount acquisition unit 302 acquires the change amount between the position of the user's visual organ and the position of the visual organ that has changed over time from that position. Here, the relative speed acquisition unit 302b of the change amount acquisition unit 302 acquires the relative speed as the change amount. For example, the relative speed v is acquired from the position (Xs1, Ys1) within the image of the user's lacrimal caruncle detected from the captured image of a certain frame (frame N) and the position (Xs0, Ys0) within the image of the lacrimal caruncle detected from the captured image of the previous frame (frame N - 1). The relative speed v is obtained by the following formula (3). v={(Xs1 - Xs0) 2 +(Ys1 - Ys0) 2} 1 / 2 ···(3)

[0047] In step S1009, the control unit 211 increments the number of times Lost_count that the visual organ position has been lost.

[0048] In step S1010, the relative speed acquisition unit 302b performs subsequent control by setting as follows in formulas (4) and (5) respectively. d = d_max ···(4) v = v_max ···(5) Here, d_max and v_max are set to large values that cannot be calculated by normal operations.

[0049] In step S1011, the control unit 211 calculates the moving average of the relative distance d. The moving average value MA_d1 of the relative distance d from the current frame (frame N) for the past m and the moving average value MA_d2 of the relative distance d for the past n are obtained by the following formulas (6) and (7) respectively. MA_d1 = Σ N N-m d / m ···(6) MA_d2 = Σ N N-n d / n ···(7)

[0050] Here,[[]]ID=41]] m > n ···(8) is true.​

[0051] In step S1012, the control unit 211 calculates the integrated value of the relative velocity v. The integrated value S_v1 of the past m relative velocities v from the current frame (frame N), and the integrated value S_d2 of the past n relative velocities v from the past are obtained by equations (9) and (10) below, respectively. S_v1=Σ N N-m v ···(9) S_v2=Σ N N-n v ···(10)

[0052] In step S1013, the image quality determination unit 303 of the control unit 211 determines the quality of the display image shown in the image display unit 203. The process in step S1013 will be explained below with reference to Figure 7. Figure 7 is a state transition diagram showing the method for determining the quality of the display image. It shows each transition condition from state S0 to S3 (t_01, t_10, t_12, t_21, t_x3, t_30) and the operation in each state.

[0053] When the image quality determination process begins, the state becomes S0. In state S0, the image quality determination unit 303 first determines whether the transition condition t_x3 to state S3 is met, and if it is determined that the transition condition to state S3 is met, it transitions to state S3. Here, the transition condition t_x3 is determined, for example, by the following equation (11). Lost_count>L_th ···(11)

[0054] Thus, in equation (11), it is determined whether the transition condition t_x3 is satisfied by whether or not Lost_count exceeds the threshold L_th. If it is determined that Lost_count exceeds the threshold L_th, it is determined that the transition condition t_x3 is satisfied and the system transitions to state S3. If it is determined that Lost_count is less than or equal to the threshold L_th, it is determined that the transition condition t_x3 is not satisfied. In this case, the image quality determination unit 303 determines whether or not the transition condition t_01 from state S0 to state S1 is satisfied, and if it is determined that the transition condition is satisfied, the system transitions to state S1. Here, the transition condition t_01 is determined, for example, by the following equation (12). MA_d1>d_th1 ···(12)

[0055] Thus, in equation (12), whether or not the transition condition t_01 is satisfied is determined by whether or not MA_d1 obtained in equation (6) reaches the threshold d_th1. If it is determined that MA_d1 exceeds the threshold d_th1, it is determined that the transition condition t_01 is satisfied. If it is determined that MA_d1 is less than or equal to the threshold d_th1, the state remains S0.

[0056] In state S1, the image quality determination unit 303 first determines whether the transition condition t_x3 to state S3 is met. If it is determined that the transition condition t_x3 is met, it transitions to state S3. If it is determined that the transition condition t_x3 is not met, the image quality determination unit 303 determines whether the transition condition t_10 to state S0 is met. If it is determined that the transition condition t_10 is met, it transitions to state S0. Here, the transition condition t_10 is determined, for example, by equation (13) below. MA_d2≦d_th2 ···(13)

[0057] Thus, in equation (13), it is determined whether the transition condition t_10 is satisfied by whether MA_d2 obtained in equation (7) is less than or equal to the threshold d_th2. If MA_d2 is less than or equal to d_th2, it is determined that the transition condition t_10 is satisfied, and the system transitions to state S0. When a transition from state S1 to state S0 occurs, the image quality determination unit 303 initializes the relative distance d, which has been accumulated for each frame, to 0.

[0058] Here, as shown in equation (8), m>n Therefore, the transition from state S0 to state S1 is less likely to occur than the transition from state S1 to state S0. When in state S0, if the HMD body 100 remains away from the user's face for a period of time, it will transition from state S0 to state S1. Conversely, when in state S1, as soon as the HMD body 100 approaches the user's face, it will transition from state S1 to state S0.

[0059] If MA_d2 is greater than d_th2 in equation (13), the image quality determination unit 303 determines whether the transition condition t_12 to state S2 is satisfied, and if it is determined that the transition condition t_12 is satisfied, it transitions to state S2. Here, the transition condition t_12 is determined, for example, by the following equation (14). S_v1>v_th1 ···(14)

[0060] Thus, in equation (14), whether or not the transition condition t_12 is satisfied is determined by whether or not S_v1 obtained in equation (7) exceeds the threshold v_th1. If it is determined that S_v1 is greater than v_th1, it is determined that the transition condition t_12 is satisfied, and the state transitions to S2. If it is determined that S_v1 is less than or equal to v_th1, the state remains in S1.

[0061] In state S2, the image quality determination unit 303 first determines whether the transition condition t_x3 to state S3 is met. If it determines that the transition condition t_x3 is met, it transitions to state S3. If the transition condition to state S3 is not met, the image quality determination unit 303 determines whether the transition condition t_21 to state S1 is met. If the transition condition t_21 is met, it transitions to state S1. Here, the transition condition t_21 is determined, for example, by equation (15) below. S_v2≦v_th2 ···(15)

[0062] Thus, in equation (15), it is determined whether the transition condition t_21 is satisfied by whether S_v2 obtained in equation (10) is less than or equal to the threshold v_th2. If it is determined that S_v2 is less than or equal to the threshold v_th2, it is determined that the transition condition t_21 is satisfied, and the system transitions to state S1. When a transition from state S2 to state S1 occurs, the image quality determination unit 303 initializes the relative velocity v, which has been accumulated for each frame, to 0. If it is determined that S_v2 is greater than the threshold v_th2, the system remains in state S2.

[0063] Here, as shown in equation (8), m>n Therefore, the transition from state S1 to state S2 is less likely to occur than the transition from state S2 to state S1. When in state S1, if the distance between the HMD body 100 and the user's face fluctuates significantly for a while, the system will transition from state S1 to state S2. In contrast, when in state S2, the system will transition from state S2 to state S1 as soon as the fluctuation in the distance between the HMD body 100 and the user's face decreases.

[0064] In state S3, the image quality determination unit 303 determines whether the transition condition t_30 to state S0 is met, and if it is determined that the transition condition t_30 is met, it transitions to state S0. Here, the transition condition t_30 is determined, for example, by equation (16) below. Here, d and v are the relative distance and relative velocity in the current frame (frame N). (d≦d_th3)&(v≦v_th3) ···(16)

[0065] Thus, the transition condition t_30 is determined by whether or not equation (16) is satisfied. If it is determined that equation (16) is satisfied, it is determined that the transition condition t_30 is satisfied, and the system transitions to state S0. When a transition from state S3 to state S0 occurs, the image quality determination unit 303 initializes the accumulated relative distance d and relative velocity v to 0. If it is determined that equation (16) is not satisfied, the system remains in state S3.

[0066] Returning to Figure 6, in step S1014, the image quality changing unit 304 changes the quality of the display image shown on the image display unit 203 to a quality corresponding to the state determined in step S1013. For example, the quality of the display image may be changed by altering the resolution or frame rate of the image display unit 203, or by altering the brightness of the image display unit 203. Alternatively, the resolution or frame rate of the image captured by the imaging unit 202 may be changed. Furthermore, the resolution of the CG displayed on the image display unit 203 may also be changed.

[0067] In some HMD systems, an imaging device (camera) is provided to measure the position and orientation information of the HMD body 100 in real space (the external environment of the information processing device 110). For example, methods such as Visual SLAM (Simultaneous Localization and Mapping) can be used. In such cases, the quality of the display image of the camera used to obtain the position and orientation information is not changed. This is because images taken with the camera in the past are used to estimate the position and orientation, and changing the quality of the display image would affect the estimation accuracy.

[0068] By changing the quality of the display image shown on the image display unit 203, the processing load on the control unit 211, HMD control unit 201, imaging unit 202, image display unit 203, etc., can be reduced, thereby reducing power consumption. It is also possible to reduce the amount of data between the control unit 211 and the HMD control unit 201.

[0069] In Figure 6, the image quality mode in state S0 provides the highest display image quality, while the quality decreases in the order of states S1, S2, and S3. As a specific example of controlling the display image quality, for example, in state S0, the resolution and frame rate of the image display unit 203 are set to the highest values. The resolution and frame rate of the image captured by the imaging unit 202 are also set to the highest values. In state S1, the resolution of the image display unit 203 is reduced, and the frame rate is also reduced. The resolution and frame rate of the image captured by the imaging unit 202 are also reduced. In state S2, in addition to the changes in state S1, the resolution of the CG is reduced. In state S3, in addition to the changes in state S1, the brightness of the image display unit 203 is reduced. Furthermore, the rendering of the CG is stopped.

[0070] If the HMD is worn on the user's head in a head-mounted style, it is easier to transition to state S0. If the user is walking around with the HMD in one hand while experiencing XR, it is easier to transition to state S1 or S2. Also, if the user takes the HMD away from their face and temporarily stops the XR experience, it transitions to state S3. Here, if the quality of the current displayed image is reduced, for example, if the image quality mode is other than state S0, the user may be notified of a warning. Specifically, the control unit 211 can display a warning message or icon on the image display unit 203, or a warning sound or voice message can be played using an audio output device (not shown). This allows the user to recognize the usage status of the HMD and adjust the wearing state as needed.

[0071] Once the process in step S1014 is complete, the process returns to step S1004 and is repeated.

[0072] According to this embodiment, an information processing device 101 is realized that can reduce power consumption without lowering the value of the user's video experience by determining the quality of the displayed image according to the user's usage conditions.

[0073] -Other Embodiments- In the embodiments described above, a computer program for controlling the imaging system is stored in a storage medium such as the memory unit 305 of the control unit 211. This computer program is a control program for realizing various functions of the CPU of the control unit 211. Specifically, each step corresponds to the program as shown in Figures 6 and 7. For example, steps S1001 to S1014 correspond to Figure 6, and various steps such as the determination of each transition condition correspond to Figure 7. The CPU of the camera control unit 211, acting as a computer, reads the computer program from the storage medium such as the memory unit 305 and executes it. The embodiments can also be realized by supplying this computer program to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The program code read from the recording medium itself realizes the functions of the embodiments described above, and the recording medium on which the program code is recorded constitutes the present disclosure.

[0074] While preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0075] The disclosure of various embodiments includes the following configurations, methods, and programs. (Composition 1) An information processing device for displaying a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, A position acquisition unit that acquires the position of the aforementioned visual organs, A change amount acquisition unit acquires the amount of change between the first position and the second position of the visual organ acquired by the position acquisition unit, A picture quality determination unit determines the quality of the displayed image according to the amount of change acquired by the amount of change acquisition unit, Having, Information processing device. (Configuration 2) A quality changing unit that changes the quality of the displayed image to the quality determined by the quality determination unit. It further has, The information processing device described in Configuration 1. (Composition 3) The aforementioned image quality changing unit is The resolution, brightness, frame rate, image processing calculation period, and the toggle of image processing on or off of the displayed image are changed. The information processing device described in Configuration 2. (Composition 4) The aforementioned change amount acquisition unit, A relative distance acquisition unit acquires the relative distance between the first position, which is the initial position of the visual organ, and the second position, which is the position of the visual organ after moving from the first position, as the amount of change. Having, An information processing device as described in one of configurations 1 to 3. (Composition 5) The initial position is stored. The information processing device described in Configuration 4. (Composition 6) The aforementioned change amount acquisition unit, A relative velocity acquisition unit that acquires the relative velocity between the first position and the second position which changes from the first position over time as the amount of change. Having, An information processing device as described in one of configurations 1 to 5. (Composition 7) The aforementioned image quality determination unit, If the amount of change remains above a threshold for a first period while operating in the first image quality mode, the system transitions to a second image quality mode, which has a lower quality display image than the first image quality mode. If the change amount remains below the threshold for a second period while operating in the second image quality mode, the system transitions to the first image quality mode. An information processing device as described in any one of configurations 1 to 6. (Composition 8) The second period is shorter than the first period. The information processing device described in Configuration 7. (Composition 9) While the quality of the displayed image is being degraded, a warning is issued to the user. An information processing device described in any one of configurations 1 to 8. (Method 1) An information processing method for displaying a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, The first step is to obtain the position of the aforementioned visual organs, A second step involves obtaining the amount of change between the first and second positions of the visual organs obtained in the first step, A third step in which the quality of the displayed image is determined according to the amount of change obtained in the second step, Having, Information processing methods. (Program 1) A computer for an information processing device that displays a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, A position acquisition unit that acquires the position of the aforementioned visual organs, A change amount acquisition unit acquires the amount of change between the first position and the second position of the visual organ acquired by the position acquisition unit, A quality determination unit determines the quality of the displayed image according to the amount of change acquired by the amount of change acquisition unit. A program designed to function as such. [Explanation of Symbols]

[0076] 100 HMD units 201 HMD Control Unit 202 Imaging Department 203 Image display section 204 Visual organ detection unit 211 Control Unit 212 Content Database 301 Position acquisition part 302 Change amount acquisition unit 302a Relative distance acquisition unit 302b Relative velocity acquisition unit 303 Image Quality Determination Section 304 Image Quality Adjustment Section 305 Storage section

Claims

1. An information processing device for displaying a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, A position acquisition unit that acquires the position of the aforementioned visual organs, A change amount acquisition unit acquires the amount of change between the first position and the second position of the visual organ acquired by the position acquisition unit, A picture quality determination unit determines the quality of the displayed image according to the amount of change acquired by the amount of change acquisition unit, Having, Information processing device.

2. A quality changing unit that changes the quality of the displayed image to the quality determined by the quality determination unit. It further has, The information processing apparatus according to claim 1.

3. The aforementioned image quality changing unit is The resolution, brightness, frame rate, image processing calculation period, and the toggle of image processing on or off of the displayed image are changed. The information processing apparatus according to claim 2.

4. The aforementioned change amount acquisition unit, A relative distance acquisition unit acquires the relative distance between the first position, which is the initial position of the visual organ, and the second position, which is the position of the visual organ after moving from the first position, as the amount of change. Having, The information processing apparatus according to claim 1.

5. The initial position is stored. The information processing apparatus according to claim 4.

6. The aforementioned change amount acquisition unit, A relative velocity acquisition unit that acquires the relative velocity between the first position and the second position which changes from the first position over time as the amount of change. Having, The information processing apparatus according to claim 1.

7. The aforementioned image quality determination unit, If the amount of change remains above a threshold for a first period while operating in the first image quality mode, the system transitions to a second image quality mode, which has a lower quality display image than the first image quality mode. If the change amount remains below the threshold for a second period while operating in the second image quality mode, the system transitions to the first image quality mode. The information processing apparatus according to claim 1.

8. The second period is shorter than the first period. The information processing apparatus according to claim 7.

9. While the quality of the displayed image is being degraded, a warning is issued to the user. The information processing apparatus according to claim 1.

10. An information processing method for displaying a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, The first step is to obtain the position of the visual organ, A second step involves obtaining the amount of change between the first and second positions of the visual organs obtained in the first step, A third step involves determining the quality of the display image according to the amount of change obtained in the second step, Having, Information processing methods.

11. A computer for an information processing device that displays a virtual space image or a composite image of a real space image and a virtual space image as a display image in the vicinity of the visual organs, A position acquisition unit that acquires the position of the aforementioned visual organs, A change amount acquisition unit acquires the amount of change between the first position and the second position of the visual organ acquired by the position acquisition unit, A quality determination unit determines the quality of the displayed image according to the amount of change acquired by the amount of change acquisition unit. A program designed to function as such.

Images