Information processing device, method for controlling the information processing device, and program

Abstract

The objective is to provide an information processing device, a control method for the information processing device, and a program that can adaptively reduce power consumption. [Solution] The information processing device 100 includes a virtual space display range acquisition means 205 that acquires the range of a virtual space 403 displayed on the display unit 109 by superimposing it with the real space 402 captured by the background camera of the imaging unit 106 as a virtual space display range, and a camera drive switching means 206 that switches the driving method of the background camera of the imaging unit 106 according to the size of the virtual space display range.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for an information processing apparatus, and a program.

Background Art

[0002] In recent years, the accuracy of head-mounted displays (hereinafter referred to as "HMDs") that can experience mixed reality (hereinafter referred to as "MR") has been improving. Along with this, cameras that acquire images of the real space (hereinafter referred to as "real space images") displayed as background images in MR operate at high frame rates and high resolutions. Therefore, in an HMD, since processing of real space images at high frame rates and high resolutions is performed, more power is consumed. For this reason, in order for a user to use a stand-alone HMD equipped with a mobile battery for a long time, a mechanism for suppressing power consumption is required.

[0003] As a mechanism for suppressing power consumption, for example, it is conceivable to reduce the frame rate and resolution when causing the camera to acquire a real space image. However, in MR, since an image of a virtual space (hereinafter referred to as "virtual space image") is overlaid and displayed on the background image, if the quality of the background image deteriorates as a result of reducing the frame rate and resolution of the camera, the quality of the user experience will also deteriorate. On the other hand, Patent Document 1 discloses a technique for controlling so as to suppress the power consumption of a camera mounted on an HMD when the imaging direction of the camera exits from a predetermined angular range.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, users experiencing MR with an HMD can not only view in various directions but also freely change the display mode of the MR image. In this case, if power saving is controlled within a predetermined angle range, as in the technology disclosed in Patent Document 1, it becomes difficult to achieve power saving that matches the display mode of the MR image.

[0006] This invention has been made in view of the above-mentioned problems. The object of this invention is to provide an information processing device, a control method for the information processing device, and a program that can adaptively reduce power consumption. [Means for solving the problem]

[0007] To achieve the above objective, the information processing device of the present invention is characterized by comprising: a first acquisition means for acquiring the range of a virtual space displayed on a display unit by superimposing it with the real space captured by a first camera as a display range; and a camera drive switching means for switching the driving method of the first camera according to the size of the display range. [Effects of the Invention]

[0008] According to the present invention, power consumption can be adaptively reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the hardware configuration of the information processing device according to the first embodiment. [Figure 2] This figure shows an example of the software logic configuration of the information processing device of the first embodiment. [Figure 3] This flowchart shows the flow of switching the driving method of the background camera in the first embodiment. [Figure 4] This figure illustrates the relationship between the virtual space display range and the switching method of driving the background camera in the first embodiment. [Figure 5]This figure shows a pop-up or icon used to notify the user that the background camera is in a power-saving state, according to the first embodiment. [Figure 6] This figure shows a user interface ("UI") screen for the user to instruct the user to change the default size setting in the first embodiment. [Figure 7] This figure shows an example of the software logic configuration of the information processing device of the second embodiment. [Figure 8] This flowchart shows the flow of events when the background camera driving method is switched according to the second embodiment. [Figure 9] This figure illustrates the relationship between the virtual space display range and the switching method of driving the background camera in the second embodiment. [Figure 10] This figure shows a UI screen for the user to instruct the user to change the default first transparency setting and the default number of pixels setting in the second embodiment. [Figure 11] This figure shows an example of the software logic configuration of the information processing device according to the third embodiment. [Figure 12] This flowchart shows the flow of events when the background camera driving method is switched according to the third embodiment. [Figure 13] In the third embodiment, this figure shows a situation where the user's hand overlaps with the virtual space when the background camera is operating in power-saving mode, from an overhead view above the user. [Figure 14] This figure shows one specific example of displaying a person other than the user in the third embodiment. [Modes for carrying out the invention]

[0010] The embodiments of the present invention will now be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can perform a similar function. In addition, any additional components may be added. Furthermore, any two or more configurations (features) from each embodiment can be combined. Moreover, not all combinations of features described in each embodiment are essential to the solution of the present invention. In addition, the features of each embodiment may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (operating conditions, operating environment, etc.).

[0011] <First Embodiment> The first embodiment will be described below with reference to Figures 1 to 6. Figure 1 is a diagram showing an example of the hardware configuration of the information processing device 100 of the first embodiment. As shown in Figure 1, the information processing device 100 has a CPU 101, ROM 102, RAM 103, power control unit 104, bus 105, imaging unit 106, distance sensor 107, microphone 108, and display unit 109. In the information processing device 100, these hardware components (excluding the bus 105) are connected to the bus 105 so that they can exchange data with each other. The CPU 101 is an arithmetic unit that comprehensively controls the information processing device 100 and performs various processing by executing various programs stored in the ROM 102.

[0012] ROM 102 is a read-only non-volatile memory device that stores various programs and parameters that do not require modification. RAM 103 is a memory device that provides a workspace to the CPU 101 and temporarily stores input information from various devices and calculation results in image processing. For example, real-world images, rendered virtual space images, and position and orientation data of the HMD described later are stored in RAM 103. Power control unit 104 is connected to the power supply that supplies power to the imaging unit 106 of the information processing device 100 and performs power supply control and management of remaining power capacity.

[0013] The imaging unit 106 is an imaging device incorporated in the information processing apparatus 100. The imaging device includes a background camera (first camera) for background display of the MR image, a position camera (second camera) for estimating the position and posture of the user's hand described later, and the like. The distance sensor 107 (third camera) is a sensor capable of measuring distance. Examples of the distance sensor 107 include a LiDAR and a TOF sensor. The microphone 108 is a device that collects sounds around the information processing apparatus 100. The display unit 109 is a display device capable of displaying an MR image, a VR image, or the like.

[0014] The information processing apparatus 100 having the above-described hardware components is configured to be incorporated in a video see-through type goggle-type or glasses-type HMD worn on the user's head. Note that the information processing apparatus 100 may be incorporated in, for example, a smartphone, a tablet, or a digital camera in addition to the HMD. Further, an information processing system may be constructed by connecting the information processing apparatus 100 having the CPU 101, the ROM 102, and the RAM 103 to an HMD having the power control unit 104, the imaging unit 106, the distance sensor 107, the microphone 108, and the display unit 109.

[0015] FIG. 2 is a diagram showing an example of the software logic configuration of the information processing apparatus 100 according to the first embodiment. The background image acquisition means 201 acquires a real space image from the background camera of the imaging unit 106 as an MR background image. The virtual space image acquisition means 202 acquires a virtual space image in which a CG object in the virtual space is rendered. The image composition means 203 composes the background image obtained from the background image acquisition means 201 and the virtual space image obtained from the virtual space image acquisition means 202 to acquire a composite image. The power control means 204 notifies the drive power supply and the camera drive switching means 206 described later of the remaining battery capacity.

[0016] The virtual space display range acquisition means 205 (first acquisition means) acquires the range of the virtual space currently displayed on the display unit 109 (hereinafter referred to as the "virtual space display range") (display range). The virtual space display range may be the range of the three-dimensional space occupied by the virtual space, or the range of the two-dimensional space occupied by the opaque region of the virtual space image. The virtual space display range acquisition means 205 may also acquire the virtual space display range from the display mode of the current virtual space image. The camera drive switching means 206 compares the size of the virtual space display range obtained from the virtual space display range acquisition means 205 with a predetermined size (second threshold) to determine whether or not to switch the driving method of the background camera. In this case, the camera drive switching means 206 may determine the size of the virtual space display range from the surface area or volume of the three-dimensional virtual space, from the surface area ratio or volume ratio of the virtual space and the real space, or from the number of pixels in the opaque region of the virtual space image. When the camera drive switching means 206 switches the driving method of the background camera, it instructs the background camera to switch the driving method.

[0017] The virtual space display range threshold setting means 207 (second setting change means) changes the default size setting used by the camera drive switching means 206 in response to instructions from the UI screen. The power saving notification means 208 creates notification information when the camera drive switching means 206 switches the driving method of the background camera so that the background camera operates in power saving mode. A detailed explanation of power saving mode will be given later. The display means 209 displays the composite image obtained from the image synthesis means 203 as an MR image on the display unit 109. The display means 209 further displays the notification information obtained from the power saving notification means 208 on the display unit 109 as a pop-up or icon. This notifies the user that the background camera is operating in power saving mode (hereinafter referred to as "power saving state"). The power saving notification means 208 may notify the user of the power saving state not only through the display of pop-ups or icons by the display means 209, but also through methods such as lighting, blinking, or sound.

[0018] Figure 3 is a flowchart showing the flow when the background camera driving method is switched in the first embodiment. The flowchart in Figure 3 (control method of the information processing device) is realized when the CPU 101 (computer) reads the program stored in ROM 102, loads it into RAM 103, and executes it. Note that the flowchart in Figure 3 may be executed for each frame of the real-world image acquired by the background camera, or it may be executed periodically at a specific time interval. Furthermore, the flowchart in Figure 3 may be executed triggered by a change in the virtual space display range. These points are also true for the flowcharts in Figures 8 and 12 described later.

[0019] When the flowchart in Figure 3 is initiated, in step S301, the CPU 101 acquires the virtual space display range using the virtual space display range acquisition means 205 (first acquisition step). In step S302, the CPU 101 determines whether the size of the virtual space display range exceeds a default size using the camera drive switching means 206. If the CPU 101 determines that the size of the virtual space display range exceeds a default size using the camera drive switching means 206, the process proceeds to step S303. On the other hand, if the CPU 101 determines that the size of the virtual space display range does not exceed a default size using the camera drive switching means 206, the process proceeds to step S305, which will be described later. As mentioned above, the CPU 101 can change the default size setting in response to instructions from the UI screen using the virtual space display range threshold setting means 207. A detailed explanation of the UI screen will be described later.

[0020] In step S303, the CPU 101 switches the driving method of the background camera by the camera driving switching means 206 so that the background camera operates in power-saving mode (camera driving switching step). Power-saving mode is a mode in which the driving power of the background camera is reduced by lowering the frame rate or resolution compared to the normal mode described later. In this case, the camera driving switching means 206 may gradually lower the frame rate or resolution of the background camera. Alternatively, the camera driving switching means 206 may operate the background camera by other power-saving driving methods instead of lowering functions such as the frame rate or resolution of the background camera.

[0021] Furthermore, the camera drive switching means 206 may switch the drive method of the background camera so that the background camera operates in power-saving mode when the remaining power capacity obtained from the power control means 204 falls below a predetermined first capacity. Alternatively, the camera drive switching means 206 may reduce the default size when the remaining power capacity obtained from the power control means 204 falls below a predetermined second capacity (default capacity). In this way, when the remaining power capacity of the power supply that provides drive power to the background camera of the information processing device 100 becomes low, the background camera is more likely to transition to power-saving mode.

[0022] In step S304, the CPU 101 notifies the user that the background camera is in a power-saving state using the power-saving notification means 208 and the display means 209. This allows the user to be aware of the decrease in the background camera's power output. The flowchart in Figure 3 then ends. The notification to the user is made, for example, by a pop-up or icon displayed on the display unit 109. A detailed explanation of the pop-up and icon will be given later.

[0023] In step S305, the CPU 101 switches the background camera's driving method using the camera driving switching means 206 so that the background camera operates in normal mode (camera driving switching step). At this time, the camera driving switching means 206 returns the background camera settings that were changed when transitioning to power saving mode back to their original settings. In other words, normal mode is a mode in which the background camera's driving power is increased by increasing the frame rate or resolution compared to power saving mode. At this time, the camera driving switching means 206 may gradually increase the frame rate and resolution of the background camera. In this way, the background camera acquires a high-quality real-world image as the background image. After that, the flowchart in Figure 3 ends.

[0024] Figure 4 is a diagram illustrating the relationship between the virtual space display range and the switching method of driving the background camera in the first embodiment. Figure 4(a) is a diagram showing an overview of the real space 402 and virtual space 403 as seen by user 401 wearing an HMD 400 with an information processing device 100 incorporated into it, from an overhead view above user 401. User 401 is wearing the HMD 400 on their head and viewing the MR image. The real space 402 appears to user 401 as the background of the MR image. The virtual space 403 appears to user 401 as a CG object of the MR image. 404 shows an example of a default size.

[0025] In this case, as described above, the virtual space display range acquisition means 205 acquires the range of the three-dimensional space occupied by the virtual space 403 as the virtual space display range. On the other hand, the range of the three-dimensional space, which combines the virtual space 403 and the real space 402, occupies most of the user 401's field of view. In this respect, the virtual space 403 is deployed so as to cover the user 401's field of view. Therefore, the range of the three-dimensional space occupied by the virtual space 403 almost completely occupies the user 401's field of view. As a result, the range in which the user 401 can see the real space 402 (i.e., the MR background image) is narrow.

[0026] Thus, when the range of the 3D space occupied by the virtual space 403, that is, the virtual space display range, is wide, even if the quality of the background image is reduced by lowering the frame rate and resolution of the background camera, the disadvantage to the user experience is minimal. Therefore, when the camera drive switching means 206 determines that the size of the virtual space display range exceeds the default size 404, it switches the drive method of the background camera so that the background camera operates in power-saving mode (step S303). This reduces the power consumption of the HMD400.

[0027] Figure 4(b) shows one specific example of how the virtual space 403 in Figure 4(a) can be displayed. In the case shown in Figure 4(b), the virtual space 403 appears to the user 401 as CG objects in an MR image that are deployed to cover the user 401's field of view. Furthermore, the display range of the CG objects in the MR image, i.e., the virtual space display range, exceeds the default size 404. For this reason, the camera drive switching means 206 switches the driving method of the background camera so that the background camera operates in power-saving mode (step S303). Note that the virtual space 403 may be deployed so that it appears to the user 401 as multiple CG objects in an MR image. Also, in Figure 4(b), CG objects are given as a specific example of how the virtual space 403 can be displayed, but a VR space or the like may also be used.

[0028] Figure 5 shows a pop-up and icon for notifying the user 401 that the background camera is in a power-saving state in the first embodiment. Figure 5(a) shows an example of a pop-up indicating that the background camera is transitioning to power-saving mode. 501 is an MR image displayed on the display unit 109. 502 is a pop-up indicating that the background camera is transitioning to power-saving mode. Figure 5(b) shows an example of an icon indicating that the background camera is transitioning to power-saving mode. 503 is a pop-up composed of a CG object. 504 is an icon indicating that the background camera is transitioning to power-saving mode, and is displayed above the pop-up 503 composed of a CG object. The power-saving notification means 208 and the display means 209 trigger the display of the pop-up 502 or icon 504 overlaid on the MR image 501 when the driving method of the background camera is switched so that the background camera operates in power-saving mode (step S304).

[0029] Figure 6 shows a UI screen for user 401 to instruct a change in the default size setting in the first embodiment. 601 is a UI screen for user 401 to instruct a change in the default size setting, and is displayed as a pop-up on the MR image 501 displayed on the display unit 109. The UI screen 601 is a UI composed of CG objects and includes a toggle switch 602 and a slider 603. The toggle switch 602 is a UI used by user 401 to switch the power saving function of the background camera on or off. The slider 603 is a UI used by user 401 to select a size. User 401 can make the UI screen 601 pop up using the HMD400 controller (not shown), etc. Furthermore, user 401 can operate the toggle switch 602 and slider 603 using the HMD400 controller, etc.

[0030] User 401 can enable the power saving function of the background camera using the toggle switch 602, and then select the size using the slider 603. Through this switching and selection operation, User 401 instructs the UI screen 601 to change the default size setting. In response to this instruction, the virtual space display range threshold setting means 207 changes the default size setting. At this time, the virtual space display range threshold setting means 207 changes the default size used for comparison with the size of the virtual space display range to the size selected by the slider 603 on the UI screen 601. In this way, User 401 can cause the information processing device 100 to change the default size setting.

[0031] If the default size is changed to a larger setting, the camera drive switching means 206 can switch the background camera's drive method so that the background camera operates in power-saving mode when the virtual space 403 is displayed on the display unit 109 over a larger area. Conversely, if the default size is changed to a smaller setting, the camera drive switching means 206 can change the background camera's drive method so that the background camera operates in power-saving mode when the virtual space 403 is displayed on the display unit 109 over a smaller area. Note that the UI used by user 401 to select the size is not limited to the slider 603, but could also be radio buttons that allow user 401 to select one of options such as "small," "normal," or "large."

[0032] Based on the above, the information processing device 100 of the first embodiment can adaptively reduce the power consumption of the HMD400 by switching the driving method of the background camera according to the range of the virtual space 403 currently displayed on the display unit 109, that is, the size of the virtual space display range.

[0033] <Second Embodiment> The second embodiment will be described below with reference to Figures 7 to 10. In the second embodiment, a method for determining whether or not to switch the background camera driving method will be described using the virtual space display range acquired taking into account the transparency of the virtual space. In the second embodiment, the same components and processes as in the first embodiment will be denoted by the same reference numerals, and their descriptions will be omitted. Therefore, the differences from the first embodiment will be described in the second embodiment.

[0034] Figure 7 shows an example of the software logic configuration of the information processing device 100 in the second embodiment. The image processing means 701 performs image processing on the composite image obtained from the image synthesis means 203. Image processing includes, for example, blurring to create an effect that blurs areas other than the area of ​​interest. The virtual space transparency acquisition means 702 (second acquisition means) acquires the transparency of the virtual space 403 (hereinafter referred to as "virtual space transparency"). In the virtual space 403, only the MR background image is displayed on the display unit 109 in the area where the virtual space transparency is maximum, and only the MR virtual space image (i.e., CG object) is displayed on the display unit 109 in the area where the virtual space transparency is minimum.

[0035] The background image processing area acquisition means 703 (third acquisition means) acquires the background image processing pixel count. The background image processing pixel count is the number of pixels in an area of ​​the background image where details have become difficult for the user 401 to see due to image processing applied to a part of the background image. Examples of image processing applied to a part of the background image include blurring. The background image processing area acquisition means 703 may also acquire the background image processing pixel count from the area corresponding to the background image in the composite image processed by the image processing means 701. Alternatively, the background image processing area acquisition means 703 may acquire the background image processing pixel count from the current background image display mode. The background image display mode is a mode that determines whether or not to perform image processing that applies some kind of image processing to the background image.

[0036] The camera drive switching means 704 uses the virtual space display range obtained from the virtual space display range acquisition means 205 and the virtual space transparency obtained from the virtual space transparency acquisition means 702 to determine whether or not to switch the background camera drive method. In this case, unlike the first embodiment, the camera drive switching means 704 defines the virtual space display range as the range of the virtual space 403 currently displayed on the display unit 109 in which the virtual space transparency is less than or equal to a predetermined first transparency (first threshold). If the size of the virtual space display range exceeds the predetermined size, the camera drive switching means 704 switches the background camera drive method so that the background camera operates in power-saving mode.

[0037] Furthermore, the camera drive switching means 704 uses the number of background image processing pixels obtained from the background image processing area acquisition means 703 to determine whether or not to switch the background camera drive method. In this case, if the number of background image processing pixels exceeds a predetermined number of pixels (third threshold), even if the size of the virtual space display range is less than or equal to a predetermined size, the camera drive switching means 704 switches the background camera drive method so that the background camera operates in power-saving mode. Conversely, if the size of the virtual space display range is less than or equal to a predetermined size and the number of background image processing pixels is less than or equal to a predetermined number of pixels, the camera drive switching means 704 switches the background camera drive method so that the background camera operates in normal mode.

[0038] The transparency threshold setting means 705 (first setting change means) changes the default first transparency setting that the camera drive switching means 704 compares with the virtual space transparency in response to instructions from the UI screen 601. The image processing pixel count threshold setting means 706 (third setting change means) changes the default pixel count setting that the camera drive switching means 704 compares with the background image processing pixel count in response to instructions from the UI screen 601.

[0039] Figure 8 is a flowchart showing the flow when the background camera driving method is switched in the second embodiment. In step S801, the CPU 101 acquires the virtual space transparency using the virtual space transparency acquisition means 702. At this time, the virtual space transparency acquisition means 702 may acquire the virtual space transparency from the transparency specified within the range of the virtual space 403 currently displayed on the display unit 109. Alternatively, the virtual space transparency acquisition means 702 may acquire the virtual space transparency from the opacity set for the virtual space image. In this case, the virtual space transparency acquisition means 702 acquires the virtual space transparency by inverting the opacity. Furthermore, if possible, the virtual space transparency acquisition means 702 may acquire the virtual space transparency from the virtual space image. In step S802, the CPU 101 acquires the background image processing pixel count using the background image processing area acquisition means 703.

[0040] In step S803, the CPU 101 uses the camera drive switching means 704 to determine whether the size of the virtual space display range exceeds a default size. However, as described above, the virtual space display range is the range of the virtual space 403 currently displayed on the display unit 109, with the virtual space transparency being less than or equal to the default first transparency. In other words, in step S803, the virtual space display range, taking virtual space transparency into account, is compared with the default size. If the CPU 101 uses the camera drive switching means 704 to determine that the size of the virtual space display range exceeds a default size, the process proceeds to step S303. On the other hand, if the CPU 101 uses the camera drive switching means 704 to determine that the size of the virtual space display range does not exceed a default size, the process proceeds to step S804. As described above, the CPU 101 can change the default first transparency setting in response to instructions from the UI screen 601 using the transparency threshold setting means 705. A detailed explanation of this point will be given later.

[0041] In step S804, the CPU 101 uses the camera drive switching means 704 to determine whether the number of pixels processed in the background image exceeds a predetermined number of pixels. If the CPU 101 determines that the number of pixels processed in the background image exceeds a predetermined number of pixels, the process proceeds to step S303. On the other hand, if the CPU 101 determines that the number of pixels processed in the background image does not exceed a predetermined number of pixels, the process proceeds to step S305. As described above, the CPU 101 can change the default number of pixels setting in response to instructions from the UI screen 601 using the image processing pixel threshold setting means 706. A detailed explanation of this will be given later.

[0042] Figure 9 is a diagram illustrating the relationship between the virtual space display range and the switching of the background camera driving method in the second embodiment. Figure 9(a) shows the virtual space display range when the background camera of the information processing device 100 incorporated in the HMD 400 worn by user 401 transitions to power saving mode, from a bird's-eye view above user 401. 901 is a portion of the virtual space 403. The portion 901 of the virtual space 403 is at the boundary with the real space 402.

[0043] In portion 901 of the virtual space 403, the real space 402 is displayed transparently through the virtual space 403. However, since the transparency of portion 901 of the virtual space 403 is lower than the default first transparency, the display ratio of the virtual space 403 is higher than the display ratio of the real space 402 which forms the background of the MR image 501. Therefore, even if the quality of the background image is reduced due to a decrease in the frame rate and resolution of the background camera, the impact on the user experience is minimal in portion 901 of the virtual space 403. Accordingly, as shown in Figure 9(a), if the size of the range of the virtual space 403 including portion 901 exceeds the default size 405, the camera drive switching means 206 switches the drive method of the background camera so that the background camera operates in power-saving mode (step S303). This reduces the power consumption of the HMD 400.

[0044] Figure 9(b) shows the virtual space display range when the background camera of the information processing device 100, which is built into the HMD 400 worn by user 401, transitions to normal mode, from a bird's-eye view above user 401. 902 is a portion of the virtual space 403. The portion 902 of the virtual space 403 is at the boundary with the real space 402. In portion 902 of the virtual space 403, the real space 402 is displayed transparently through the virtual space 403. However, since the virtual space transparency of portion 902 of the virtual space 403 is greater than the default first transparency, the display ratio of the real space 402, which is the background of the MR image 501, is higher than the display ratio of the virtual space 403. For this reason, if the frame rate or resolution of the background camera is reduced, the quality of the background image in portion 901 of the virtual space 403 will have a significant impact on the user experience. Therefore, as shown in Figure 9(b), the camera drive switching means 206 switches the background camera drive method so that the background camera operates in normal mode if the size of the virtual space 403 excluding portion 902 does not exceed a predetermined size 405 (step S305).

[0045] Figure 10 shows a UI screen 601 in the second embodiment for user 401 to instruct the user to change the default first transparency setting and the default number of pixels setting. The UI screen 601 is a UI composed of CG objects and has sliders 1001 and 1002 in addition to the toggle switch 602 and slider 603 described above. Slider 1001 is a UI used by user 401 when selecting transparency. Slider 1002 is a UI used by user 401 when selecting the number of pixels. User 401 can operate sliders 1001 and 1002 with the controller of the HMD400 or the like.

[0046] User 401 can enable the power saving function of the background camera using the toggle switch 602, then select transparency using the slider 1001 and select the number of pixels using the slider 1002. Through these switching and selection operations, User 401 can instruct the UI screen 601 to change the default first transparency setting and the default number of pixels setting. In response to this instruction, the transparency threshold setting means 705 changes the default first transparency setting, and the image processing pixel count threshold setting means 706 changes the default number of pixels setting. At this time, the transparency threshold setting means 705 changes the default first transparency used for comparison with the virtual space transparency to the transparency selected by the slider 1001 on the UI screen 601. Also, the image processing pixel count threshold setting means 706 changes the default number of pixels used for comparison with the background image processing pixel count to the number of pixels selected by the slider 1001 on the UI screen 601. In this way, user 401 can instruct the information processing device 100 to change the default first transparency and the default number of pixels.

[0047] Here, if the default first transparency is set to a larger value, the size of the virtual space display range used for the determination in step S803 becomes larger, making it easier for the background camera to transition to power-saving mode. Also, if the default number of pixels is set to a smaller value, the size of the default number of pixels used for the determination in step S804 also becomes smaller, making it easier for the background camera to transition to power-saving mode.

[0048] As described above, the information processing device 100 of the second embodiment can more adaptively reduce the power consumption of the HMD400 by switching the driving method of the background camera according to the size of the virtual space display range acquired taking into account the transparency of the virtual space. Furthermore, if the size of the virtual space display range acquired taking into account the transparency of the virtual space does not exceed a predetermined size, the information processing device 100 of the second embodiment switches the driving method of the background camera according to the number of pixels processed in the background image. As a result, the information processing device 100 of the second embodiment can more adaptively reduce the power consumption of the HMD400.

[0049] <Third Embodiment> The third embodiment will be described below with reference to Figures 11 to 14. In the third embodiment, a method for determining whether or not to switch the background camera driving method will be described, taking into account the detection results of user 401's hand and the detection results of people other than user 401. In the third embodiment, the same components and processes as in the second embodiment will be denoted by the same reference numerals, and their descriptions will be omitted. Therefore, the differences from the second embodiment will be described in the third embodiment.

[0050] Figure 11 shows an example of the software logic configuration of the information processing device 100 according to the third embodiment. The surrounding image acquisition means 1101 causes the position camera of the imaging unit 106 to image the area around the user 401, thereby acquiring images from the position camera to estimate the position and orientation of the user 401's hands. The hand detection means 1102 detects the hands from the images obtained from the surrounding image acquisition means 1101. The hand region extraction means 1103 (first extraction means) uses the hand detection result obtained from the hand detection means 1102 to extract the image region of the hands in the background image.

[0051] The 3D distance acquisition means 1104 acquires a 3D distance image of the real space 402 from the distance sensor 107. The sound acquisition means 1105 acquires sound from the microphone 108, that is, sound from the surroundings of the information processing device 100, i.e., sound from the surroundings of the user 401 wearing the HMD 400. The person detection means 1106 detects people within a predetermined distance from the distance sensor 107, i.e., people within a predetermined distance from the user 401 wearing the HMD 400, from the 3D distance image obtained from the 3D distance acquisition means 1104. In this case, the person detection means 1106 may take into account the sound obtained from the sound acquisition means 1105 and narrow down the detection to people who are speaking to the user 401 among the people within a predetermined distance from the distance sensor 107. The person region extraction means 1107 (second extraction means) extracts the image region of a person in the background image using the person detection result obtained from the person detection means 1106.

[0052] The image synthesis means 1108 takes into account the hand image region obtained from the hand region extraction means 1103 and the person image region obtained from the person region extraction means 1107, and synthesizes the background image and the virtual space image to obtain a composite image. In this case, the image synthesis means 1108 may synthesize the images so that the hands and people in the background image are clearly displayed by maximizing the virtual space transparency of the parts of the virtual space 403 that overlap with the hand image region and the person image region. The hand region virtual space transparency acquisition means 1109 (fourth acquisition means) acquires the virtual space transparency (transparency of the virtual space part) of the parts of the virtual space 403 that overlap with the hand image region. The person region virtual space transparency acquisition means 1110 (fifth acquisition means) acquires the virtual space transparency (transparency of the virtual space part) of the parts of the virtual space 403 that overlap with the person image region.

[0053] The camera drive switching means 1111 differs from the camera drive switching means 704 in the following respects. The camera drive switching means 1111 uses the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the hand or person, obtained from the hand area virtual space transparency acquisition means 1109 and the person area virtual space transparency acquisition means 1110, to determine whether or not to switch the background camera drive method. In this case, if the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the hand or person exceeds a predetermined second transparency (fourth threshold) (fifth threshold), the camera drive switching means 1111 switches the background camera drive method so that the background camera operates in normal mode. Conversely, if the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the hand or person does not exceed a predetermined second transparency, the camera drive switching means 1111 switches the background camera drive method according to the size of the virtual space display range and the number of pixels processed in the background image.

[0054] Figure 12 is a flowchart showing the flow when the background camera driving method is switched in the third embodiment. In step S1201, the CPU 101 detects hands and people using the hand detection means 1102 and the person detection means 1106. In step S1202, the CPU 101 extracts the image regions of hands and people using the hand region extraction means 1103 and the person region extraction means 1107. In step S1203, the CPU 101 acquires the virtual space transparency of the portion of the virtual space 403 that overlaps with the hand image region using the hand region virtual space transparency acquisition means 1109. Furthermore, the CPU 101 acquires the virtual space transparency of the portion of the virtual space 403 that overlaps with the person image region using the person region virtual space transparency acquisition means 1110.

[0055] In step S1204, the CPU 101 uses the camera drive switching means 1111 to determine whether the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the hand or person exceeds the default second transparency. If the CPU 101 uses the camera drive switching means 1111 to determine whether the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the hand exceeds the default second transparency, the process proceeds to step S305. Similarly, if the CPU 101 uses the camera drive switching means 1111 to determine whether the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the person exceeds the default second transparency, the process proceeds to step S305. If the CPU 101 uses the camera drive switching means 1111 to determine (1) or (2) below, the process proceeds to step S803. (1) The virtual space transparency of the portion of virtual space 403 that overlaps with the hand image area does not exceed the default second transparency. (2) The virtual space transparency of the portion of virtual space 403 that overlaps with the image area of ​​the person shall not exceed the default second transparency.

[0056] Furthermore, the CPU 101 (fourth setting change means) (fifth setting change means) may change the default second transparency setting in the same manner as the default first transparency setting change described above, in response to instructions from the UI screen 601. Also, the default second transparency that is compared with the virtual space transparency of the part of virtual space 403 that overlaps with the hand image area and the default second transparency that is compared with the virtual space transparency of the part of virtual space 403 that overlaps with the person image area may be different.

[0057] Figure 13 shows a view from above the user 401 in the third embodiment, where the user 401's hand 1301 overlaps with the virtual space 403 when the background camera is operating in power-saving mode. The relationship between the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the user 401's hand 1301 and the switching of the background camera's driving method will be explained below using Figure 13. The camera driving switching means 1111 switches the background camera's driving method so that the background camera operates in normal mode if the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the user 401's hand 1301 exceeds a predetermined second transparency (step S305). In this way, when the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the user 401's hand 1301 in the background image is large, the image area of ​​the user 401's hand 1301 in the background image is clearly displayed in the MR image 501.

[0058] In contrast, if the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the user 401's hand 1301 in the background image is small, then in the MR image 501, it is not necessary for the image area of ​​the user 401's hand 1301 in the background image to be displayed clearly. For this reason, the camera drive switching means 1111 does not switch the driving method of the background camera in the cases of (1) and (2) described above (step S1204: NO). Furthermore, if the size of the virtual space display range is not less than or equal to a predetermined size, or if the number of pixels processed in the background image is not less than or equal to a predetermined number of pixels, the camera drive switching means 1111 will operate the background camera in power-saving mode (step S303). In this way, in the MR image 501, the image area of ​​the user 401's hand 1301 in the background image, which has reduced quality, is blended with the virtual space 403 and displayed, thus reducing the power consumption of the HMD 400.

[0059] Figure 14 is a diagram showing one specific example of displaying a person other than user 401 in the third embodiment. 1401 represents the image area of ​​the person in the background image. In the MR image 501, the virtual space 403 is displayed excluding the image area 1401 of the person. The relationship between the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area 1401 of the person other than user 401 and the switching of the background camera driving method will be explained below using Figure 14. The camera driving switching means 1111 switches the driving method of the background camera so that the background camera operates in normal mode if the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area 1401 of the person exceeds the default second transparency (step S305). In this way, when the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area 1401 of the background image is large, the image area 1401 of the person in the background image is displayed clearly in the MR image 501.

[0060] In contrast, if the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area 1401 of the person in the background image is small, it is not necessary for the image area 1401 of the person in the background image to be displayed clearly in the MR image 501. For this reason, the camera drive switching means 1111 does not switch the driving method of the background camera in the cases of (1) and (2) described above (step S1204: NO). Furthermore, if the size of the virtual space display range is not less than or equal to a predetermined size, or if the number of pixels processed in the background image is not less than or equal to a predetermined number of pixels, the camera drive switching means 1111 will operate the background camera in power-saving mode (step S303). In this way, in the MR image 501, the image area 1401 of the person in the background image, which has reduced quality, is blended with the virtual space 403 and displayed, thus reducing the power consumption of the HMD 400.

[0061] Based on the above, in the third embodiment, the information processing device 100 switches the background camera's driving method so that the background camera operates in normal mode when the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area of ​​the user 401's hand 1301 exceeds the default second transparency. As a result, in the MR image 501, the user 401's hand 1301 in the real space 402 is clearly displayed, transparent to the virtual space 403. Furthermore, the information processing device 100 also switches the background camera's driving method so that the background camera operates in normal mode when the virtual space transparency of the portion of the virtual space 403 that overlaps with the image area 1401 of a person within a predetermined distance from the user 401 exceeds the default second transparency. As a result, in the MR image 501, a person within a predetermined distance from the user 401 in the real space 402 is clearly displayed, transparent to the virtual space 403.

[0062] <Other> Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. For example, the CPU 101 may change the quality of the background of the MR image 501 on which the UI screen 601 is displayed in a pop-up, according to the size selected by the slider 603. The CPU 101 may also overlay a window on the UI screen 601 displaying the background of the MR image 501 whose quality changes according to the size selected by the slider 603. The same applies to the transparency selected by the slider 1001 and the number of pixels selected by the slider 1002.

[0063] Furthermore, if the size of the virtual space display range exceeds a predetermined size, the CPU 101 may display a toggle button on the display unit 109, and when the user 401 operates the toggle button, it may proceed to step S303. The same applies when the number of pixels processed for the background image exceeds a predetermined number of pixels.

[0064] Furthermore, the CPU 101 may change the default first capacity setting and the default second capacity setting in the same manner as changing the default size setting, in response to instructions from the UI screen 601. In this case, if the CPU 101 changes the default first capacity setting or the default second capacity setting, it may notify the user of this fact through methods such as displaying a pop-up or icon by the display means 209, or by lighting, flashing, or sound. These points also apply to the default distance used by the person detection means 1106.

[0065] Furthermore, each embodiment is an example of applying the present invention to MR. However, the present invention is not limited thereto and can also be applied to Augmented Reality (AR), for example.

[0066] The present invention can also be realized by supplying a program that implements one or more of the functions of each of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a dedicated processor (for example, an ASIC, FPGA, or other circuit) that implements one or more functions. Moreover, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Note that "processor" refers to a processor in a broad sense and includes general-purpose processors and dedicated processors. Furthermore, the operation of the processor may not only be performed by a single processor, but may also be performed by multiple processors located in physically separate locations working together.

[0067] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Configuration 1) A first acquisition means that acquires the range of the virtual space displayed on the display unit by superimposing it with the real space captured by the first camera as the display range, An information processing apparatus comprising a camera drive switching means for switching the driving method of the first camera according to the size of the display range. (Configuration 2) Equipped with a second acquisition means for acquiring the transparency of the virtual space, The information processing apparatus according to Configuration 1, characterized in that the size of the display range is the size of the range in which the transparency of the virtual space acquired by the second acquisition means is less than or equal to the first threshold, among the display range. (Configuration 3) The information processing device according to Configuration 2, characterized in that it includes a first setting change means that allows the user to change the setting of the first threshold on a UI screen. (Configuration 4) The information processing apparatus according to any one of Configurations 1 to 3, characterized in that the camera drive switching means switches the driving method of the first camera so that the driving power of the first camera decreases when the size of the display range exceeds a second threshold. (Configuration 5) The information processing device according to Configuration 4, characterized in that it includes a second setting change means that allows the user to change the setting of the second threshold on a UI screen. (Configuration 6) The information processing apparatus according to Configuration 4 or 5, characterized in that the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image when the size of the display range does not exceed the second threshold. (Configuration 7) A third acquisition means is provided for acquiring the number of pixels that have undergone image processing from the image of the real space acquired by the first camera, The information processing apparatus according to configuration 6, characterized in that the camera drive switching means switches the driving method of the first camera according to the size of the display range and the number of pixels. (Configuration 8) The information processing device according to Configuration 7, characterized by comprising an image processing means for performing the image processing. (Configuration 9) The information processing apparatus according to Configuration 8, wherein the image processing means performs blurring as the image processing for the image processing. (Configuration 10) The information processing apparatus according to any one of Configurations 7 to 9, characterized in that the camera drive switching means switches the driving method of the first camera so that the driving power of the first camera decreases when the size of the display range does not exceed the second threshold or when the number of pixels exceeds the third threshold. (Configuration 11) The information processing device according to Configuration 10, characterized in that it includes a third setting change means that allows the user to change the setting of the third threshold on a UI screen. (Configuration 12) The information processing apparatus according to Configuration 10 or 11, characterized in that the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image if the size of the display range does not exceed the second threshold and the number of pixels does not exceed the third threshold. (Configuration 13) The information processing apparatus according to any one of Configurations 4 to 12, characterized in that the camera drive switching means reduces the second threshold when the remaining power capacity of the power supply that supplies drive power to the first camera is less than or equal to a predetermined capacity. (Configuration 14) A fourth acquisition means is provided for acquiring the transparency of a portion of the virtual space displayed on the display unit by superimposing it with an image area of ​​the user's hand in the real space captured by the first camera, The information processing apparatus according to any one of configurations 1 to 13, characterized in that the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image when the transparency of the portion of the virtual space acquired by the fourth acquisition means exceeds a fourth threshold. (Configuration 15) Hand detection means for detecting the hand from an image of the user's surroundings acquired by imaging with the second camera, The information processing apparatus according to configuration 14, further comprising: a first extraction means for extracting an image region of the hand from an image of the real space acquired by imaging with the first camera, using the detection result of the hand detection means. (Configuration 16) The information processing device according to Configuration 14 or 15, characterized in that it includes a fourth setting change means that allows the user to change the setting of the fourth threshold on a UI screen. (Configuration 17) A fifth acquisition means is provided for acquiring the transparency of a portion of the virtual space displayed on the display unit by superimposing it with the image area of ​​a person within a predetermined distance from a user in the real space captured by the first camera, The information processing apparatus according to any one of configurations 1 to 15, characterized in that the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image when the transparency of the portion of the virtual space acquired by the fifth acquisition means exceeds a fifth threshold. (Configuration 18) Person detection means for detecting the person from a three-dimensional distance image of the real space acquired by imaging by the third camera, The information processing apparatus according to configuration 17, further comprising: a second extraction means for extracting an image region of a person from an image of real space acquired by imaging with the first camera, using the detection result of the person detection means. (Configuration 19) The information processing device according to Configuration 18, wherein the person detection means is characterized by detecting a person who has spoken to the user based on the voice acquired by the microphone. (Configuration 20) An information processing device according to any one of Configurations 17 to 19, characterized in that it includes a fifth setting change means that allows the user to change the setting of the fifth threshold on a UI screen. (Configuration 21) An information processing device according to any one of Configurations 1 to 20, characterized in that when the camera drive switching means switches the driving method of the first camera so as to reduce the driving power of the first camera, it is provided with a power saving notification means that notifies the user of the reduction in the driving power of the first camera. (Configuration 22) The information processing apparatus according to any one of Configurations 1 to 21, characterized in that the first acquisition means acquires the display range from the display mode of the image in the current virtual space. (Configuration 23) The information processing apparatus according to any one of Configurations 1 to 21, characterized in that the first acquisition means acquires the display range from the opaque region of the image in the virtual space. (Configuration 24) The information processing device according to any one of Configurations 1 to 23, characterized in that the information processing device is a head-mounted display comprising the first camera and the display unit. (Method 1) A first acquisition step in which the range of the virtual space displayed on the display unit by superimposing it with the real space captured by the first camera is acquired as the display range, A control method for an information processing device, characterized by comprising a camera drive switching step of switching the driving method of the first camera according to the size of the display range. (Program 1) A program that causes a computer to execute each of the means of the information processing device described in any one of Configurations 1 to 24. [Explanation of symbols]

[0068] 100 Information Processing Devices 106 Imaging Unit 109 Display section 205 Virtual space display range acquisition means (first acquisition means) 206 Camera drive switching means 402 Real Space 403 Virtual Space

Claims

1. A first acquisition means acquires the range of the virtual space displayed on the display unit by superimposing it with the real space captured by the first camera as the display range, An information processing apparatus characterized by comprising a camera drive switching means for switching the driving method of the first camera according to the size of the display range.

2. It is equipped with a second acquisition means for acquiring the transparency of the virtual space, The information processing apparatus according to claim 1, characterized in that the size of the display range is the size of the range within the display range in which the transparency of the virtual space acquired by the second acquisition means is less than or equal to a first threshold.

3. The information processing device according to claim 2, further comprising a first setting change means that allows a user to change the setting of the first threshold on a UI screen.

4. The information processing apparatus according to claim 1, characterized in that the camera drive switching means switches the driving method of the first camera such that the driving power of the first camera decreases when the size of the display range exceeds a second threshold.

5. The information processing device according to claim 4, further comprising a second setting change means that allows the user to change the setting of the second threshold on a UI screen.

6. The information processing apparatus according to claim 4, wherein the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image when the size of the display range does not exceed the second threshold.

7. The system includes a third acquisition means for acquiring the number of pixels that have undergone image processing from the image of the real space acquired by the first camera, The information processing apparatus according to claim 6, wherein the camera drive switching means switches the driving method of the first camera according to the size of the display range and the number of pixels.

8. The information processing apparatus according to claim 7, characterized by comprising an image processing means for performing the aforementioned image processing.

9. The information processing apparatus according to claim 8, characterized in that the image processing means performs blurring as the image processing for the image processing.

10. The information processing apparatus according to claim 7, characterized in that the camera drive switching means switches the driving method of the first camera such that the driving power of the first camera decreases when the size of the display range does not exceed the second threshold or when the number of pixels exceeds the third threshold.

11. The information processing device according to claim 10, further comprising a third setting change means that allows the user to change the setting of the third threshold on a UI screen.

12. The information processing apparatus according to claim 10, wherein the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image if the size of the display range does not exceed the second threshold and the number of pixels does not exceed the third threshold.

13. The information processing apparatus according to claim 4, wherein the camera drive switching means reduces the second threshold when the remaining power capacity of the power supply that supplies drive power to the first camera is less than or equal to a predetermined capacity.

14. The system includes a fourth acquisition means for acquiring the transparency of a portion of the virtual space displayed on the display unit by superimposing it with an image area of ​​the user's hand in the real space captured by the first camera, The information processing apparatus according to claim 1, characterized in that the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image when the transparency of the portion of the virtual space acquired by the fourth acquisition means exceeds a fourth threshold.

15. A hand detection means for detecting the hand from an image of the user's surroundings acquired by imaging with a second camera, The information processing apparatus according to claim 14, further comprising: a first extraction means for extracting an image region of the hand from an image of the real space acquired by imaging with the first camera, using the detection result of the hand detection means.

16. The information processing device according to claim 14, further comprising a fourth setting change means that allows the user to change the setting of the fourth threshold on a UI screen.

17. The system includes a fifth acquisition means for acquiring the transparency of a portion of the virtual space displayed on the display unit by superimposing it with an image area of ​​a person within a predetermined distance from a user in the real space captured by the first camera, The information processing apparatus according to claim 1, characterized in that the camera drive switching means switches the driving method of the first camera so that the first camera acquires a high-quality image when the transparency of the portion of the virtual space acquired by the fifth acquisition means exceeds a fifth threshold.

18. A person detection means for detecting the person from a three-dimensional distance image of the real space acquired by imaging with a third camera, The information processing apparatus according to claim 17, further comprising: a second extraction means for extracting an image region of a person from an image of real space acquired by imaging with the first camera, using the detection result of the person detection means.

19. The information processing apparatus according to claim 18, characterized in that the person detection means detects the person who spoke to the user based on the voice acquired by the microphone.

20. The information processing device according to claim 17, further comprising a fifth setting change means that allows a user to change the setting of the fifth threshold on a UI screen.

21. The information processing device according to claim 1, further comprising a power saving notification means for notifying the user of the decrease in the drive power of the first camera when the camera drive switching means switches the drive method of the first camera so that the drive power of the first camera decreases.

22. The information processing apparatus according to claim 1, characterized in that the first acquisition means acquires the display range from the display mode of the image in the current virtual space.

23. The information processing apparatus according to claim 1, characterized in that the first acquisition means acquires the display range from the opaque region of the image in the virtual space.

24. The information processing apparatus according to claim 1, characterized in that the information processing apparatus is a head-mounted display comprising the first camera and the display unit.

25. A first acquisition step involves acquiring the range of the virtual space displayed on the display unit by superimposing it with the real space captured by the first camera, as the display range. A control method for an information processing device, characterized by comprising a camera drive switching step of switching the driving method of the first camera according to the size of the display range.

26. A program for causing a computer to execute each of the means of the information processing apparatus described in claim 1.

Images