Information processing device, information processing method, and program

Abstract

The present invention makes it possible to achieve both an improvement in resolution of real space images used in, for example, video perspective AR devices and MR devices and a reduction in system load. The image acquisition unit acquires a first image obtained by capturing an image at a first angle of view, and a second image obtained by capturing an image at a second angle of view that is within the first angle of view and narrower than the first angle of view. The image generation unit generates a display image by combining an image of a high-resolution region based on the second image with an image of a peripheral region in the periphery of the high-resolution region based on the first image. For example, movement of a capturing direction of a capturing unit for obtaining a second image is controlled based on gaze information of a user, and movement of a position of a high-resolution region is controlled.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more specifically, to an information processing device and others suitable for obtaining a real-space image used in, for example, a video see-through AR device, an MR device, and the like. Background Art

[0002] In recent years, the widespread adoption of virtual reality (VR) has accelerated due to low-priced virtual reality (VR) devices and abundant content. Improving resolution to reproduce reality within a space is a very effective way to improve the quality of user experience, and therefore there is a strong demand for reality with higher image quality than that of conventional two-dimensional (2D) content.

[0003] On the other hand, increasing the resolution of content and display displays increases the load on processors such as graphics processing units (GPUs) or central processing units (CPUs) and the load on systems such as memory or buses. Therefore, in order to achieve both higher resolution and reduced system load, many representative virtual reality products use foveated rendering techniques, which render the area the user is looking at with high definition and reduce the amount of rendering in peripheral areas. For example, PTL 1 describes foveated rendering.

[0004] So far, virtual reality content has generally focused on games and video viewing, but future virtual reality services will become more integrated with real society, and therefore, new experiences and services of mixed reality (MR) with augmented reality (AR) experience as a space for social and economic activities are expected to emerge. Such mixed reality is a video see-through type using camera images.

[0005] [Citation List]

[0006] [Patent Document]

[0007] [PTL1] JP 2020-042807A Summary of the invention

[0008] [Technical issues]

[0009] An object of the present technology is to achieve both a high-resolution real-space image used in, for example, a video see-through AR device or an MR device and a reduced system load.

[0010] [Solution to the problem]

[0011] The idea of ​​this technology is:

[0012] An information processing device, comprising:

[0013] an image acquisition unit that acquires a first image captured at a first angle of view and a second image captured at a second angle of view within the first angle of view that is narrower than the first angle of view; and

[0014] An image generating unit generates a display image by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image and surrounding the high-resolution area.

[0015] In the present technology, a first image captured at a first viewing angle and a second image captured at a second viewing angle narrower than the first viewing angle within the first viewing angle are acquired by an image acquisition unit. Then, an image generation unit generates a display image by synthesizing an image of a high-resolution area based on the second image with an image of a peripheral area around the high-resolution area based on the first image.

[0016] For example, the image of the high-resolution area may be an image obtained by directly using the second image, and the image of the peripheral area may be an image obtained by enlarging the first image. In this case, for example, the first image and the second image may each have a first resolution, and the image of the peripheral area may be an image having a second resolution obtained by enlarging the first image according to a ratio between the first viewing angle and the second viewing angle. Here, the first resolution may be a 1K resolution, and the second resolution may be a 4K resolution.

[0017] For example, a wide-angle image capturing unit for obtaining the first image and a zoomed-in image capturing unit for obtaining the second image may be further included. For example, a display unit for displaying the display image may be further included.

[0018] In this way, in the present technology, an image of a high-resolution area based on a first image captured at a first perspective is synthesized with an image of a peripheral area around the high-resolution area based on a second image captured at a second perspective narrower than the first perspective within the first perspective to generate a display image, which makes it possible to achieve both high-resolution real-space images used in, for example, a video see-through AR device or MR device and reduced system load.

[0019] In the present technology, for example, a control unit may be further included that controls the movement of the imaging direction of the image capture unit for obtaining the second image based on the user's gaze information, and controls the movement of the position of the high-resolution area. This makes it possible to locate the high-resolution area including the high-resolution image by following the user's gaze. In this case, for example, a gaze detection unit may be further included that detects the user's gaze information.

[0020] In the present technology, for example, a control unit may be further included that controls switching of an image capture unit for obtaining a second image based on information about the distance to a subject related to the second image. This makes it possible to provide a high-quality image with reduced blur, etc. as an image of a high-resolution area regardless of the distance to the subject.

[0021] In this case, for example, the control unit may be configured to switch the image capture unit used to obtain the second image to the first image capture unit for the first imaging distance or the second image capture unit for an imaging distance longer or shorter than the first imaging distance. Here, for example, the first image capture unit may be a normal image capture unit, and the second image capture unit may be a telephoto image capture unit or a close-up image capture unit.

[0022] In this case, for example, the control unit may be configured to switch the image capture unit used to obtain the second image to any one of: a first image capture unit for the first imaging distance, a second image capture unit for an imaging distance longer than the first imaging distance, and a third image capture unit for an imaging distance shorter than the first imaging distance. Here, for example, the first image capture unit may be a normal image capture unit, the second image capture unit may be a telephoto image capture unit, and the third image capture unit may be a close-up image capture unit.

[0023] In addition, another concept of the present technology is an information processing method, comprising the following steps:

[0024] Acquire a first image captured at a first viewing angle and a second image captured at a second viewing angle within the first viewing angle that is narrower than the first viewing angle; and

[0025] A display image is generated by synthesizing an image based on the high-resolution area of ​​the second image with an image based on the first image of a peripheral area around the high-resolution area.

[0026] In addition, another concept of the present technology is a program for causing a computer to execute an information processing method, the information processing method comprising the following steps: acquiring a first image captured at a first perspective and a second image captured at a second perspective within the first perspective that is narrower than the first perspective; and generating a display image by synthesizing an image of a high-resolution area based on the second image with an image of a peripheral area around the high-resolution area based on the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] [ Figure 1 ] Figure 1 is a diagram showing an example of a conventional VR display system.

[0028] [ Figure 2 ] Figure 2 is a diagram showing an example of a conventional AR / MR display system.

[0029] [ Figure 3 ] Figure 3 is a diagram showing a configuration example of a display system for real-space images as an embodiment.

[0030] [ Figure 4 ] Figure 4 is a diagram showing another configuration example of a display system for real-space images as an embodiment.

[0031] [ Figure 5 ] Figure 5 is a diagram showing another configuration example of a display system for real-space images as an embodiment.

[0032] [ Figure 6 ] Figure 6 is a diagram showing another configuration example of a display system for real-space images as an embodiment.

[0033] [ Figure 7 ] Figure 7 is a diagram showing another configuration example of a display system for real-space images as an embodiment.

[0034] [ Figure 8 ] Figure 8 is a configuration diagram showing an example of data transmission in a display system.

[0035] [ Fig. 9 ] Fig. 9 : is a diagram showing a configuration example (functional configuration example) of an information processing device corresponding to a display system.

[0036] [ Fig.10 ] Fig.10 is a flowchart showing an example of a processing procedure of an information processing apparatus; DETAILED DESCRIPTION

[0037] A mode for carrying out the present invention (hereinafter referred to as an "embodiment") will be described below. The description will be made in the following order.

[0038] 1. Implementation Method

[0039] 2. Modify the example

[0040] <1. Implementation Method>

[0041] [VR display system, AR / MR display system]

[0042] Figure 1An example of a conventional VR display system is shown. Drawing data is sent from an application such as a game application to a graphics processing unit (GPU) through a graphics application programming interface (API), and rendering is performed at the GPU based on the drawing data to generate a display image.

[0043] In this case, for example, in order to achieve both high resolution and reduced system load, foveated rendering may be performed in which a focus area (high resolution area) including the user's gaze point (viewpoint) and a peripheral area (low resolution area) around the focus area are set and a display image is rendered. The display image generated by this foveated rendering is displayed on the display.

[0044] Figure 2 An example of a conventional AR / MR display system is shown. A camera input as a real-space image is sent to a GPU, where a synthesis process is performed in which a user interface (UI) image and a computer graphics (CG) image are superimposed on the real-space image to generate a display image. The display image is displayed on a display.

[0045] In this case, the camera input is directly used as the real-space image in the GPU synthesis process. This makes it difficult to achieve both high-resolution real-space images and reduced system load. For example, in the case of a 4K resolution video see-through AR device or MR device, two RGB cameras (RGB 60Hz x4Kx2), two simultaneous localization and mapping (SLAM) cameras, and two displays operate simultaneously, which increases the system load.

[0046] "First configuration example of the display system"

[0047] exist Figure 3 , (a) shows a configuration example of a display system 10A for real space images as an embodiment, which is installed in a head-mounted display and used in a video see-through AR device, an MR device, etc. The configuration example shown here is only a portion for one eye. The display system 10A includes a wide-angle camera 101, a normal camera 102, a GPU 103, and a display 104.

[0048] The wide-angle camera 101 constitutes a wide-angle image capturing unit, has a wide viewing angle, and can capture a wide range image. The normal camera 102 constitutes a normal image capturing unit, has a narrow viewing angle, but can capture a narrow range image with high resolution. Since the normal camera 102 can capture a narrow range image with high resolution, it also constitutes a zoom camera. Figure 3As shown in (b), the wide-angle camera 101 captures an image at a viewing angle θ1 and outputs an image with a 1K (1080x1080) resolution, while the ordinary camera 102 captures an image at a viewing angle θ2 corresponding to 1 / 4 of the imaging range of the wide-angle camera 101 and outputs an image with a 1K (1080x1080) resolution.

[0049] GPU 103 generates a 4K resolution display image by synthesizing an image based on the focus area (high-resolution area) of the image captured by the normal camera 102 with an image based on the peripheral area (low-resolution area) around the focus area of ​​the image captured by the wide-angle camera 101 (i.e., performing foveation rendering).

[0050] In this case, as the image of the focus area (high-resolution area), the 1K resolution image captured by the normal camera 102 is directly used. As the image of the peripheral area, a 4K resolution image is used here, which is obtained by enlarging the 1K resolution image captured by the wide-angle camera 101 by a factor (i.e., 4 times) according to the ratio between the angle of view θ1 and the angle of view θ2 (i.e., the ratio of the imaging ranges of the wide-angle camera 101 and the normal camera 102).

[0051] In this case, the position of the focus area (high-resolution area) in the 4K resolution display image is set to a position within the imaging range of the wide-angle camera 101 corresponding to the position of the imaging range of the normal camera 102, and is set to the center position here.

[0052] In order to synthesize the image of the focus area (high-resolution area) and the image of the peripheral area (low-resolution area) to generate a 4K resolution display image, the GPU 103 also performs calibration processing on the joint portion between the two areas to make the joint smooth. This can reduce the discomfort felt by the user at the joint portion.

[0053] The display 104 displays a 4K resolution display image generated by the GPU 103 .

[0054] exist Figure 3 In the display system 10A shown in (a), in order to generate a 4K resolution display image, a narrow range 1K resolution image captured at a high resolution using a normal camera 102 is directly used as an image of a focus area (high resolution area), and a 4K resolution image obtained by enlarging a wide range 1K resolution image captured using a wide-angle camera 101 is used as an image of a peripheral area (low resolution area). Both a high-resolution real space image and a reduced system load can be achieved without capturing a camera input at 4K resolution.

[0055] exist Figure 3 In the display system 10A shown in (a) of FIG. 1 , an example is presented in which the wide-angle camera 101 outputs a 1K resolution image like the normal camera 102. However, the wide-angle camera 101 can also output an image with a resolution higher than 1K resolution. For example, if the wide-angle camera 101 is used to output a 2K resolution image, the image is enlarged to obtain a 4K resolution image, which is used as an image of the peripheral area.

[0056] "Example of the second configuration of the display system"

[0057] exist Figure 4 In (a), a configuration example of a display system 10B for real space images as an embodiment is shown, which is installed in a head-mounted display and used in a video see-through AR device, an MR device, etc. The configuration example shown here is only the part for one eye. Figure 4 In (a), Figure 3 The same reference numerals are used for the parts corresponding to the parts in (a), and thus the detailed description of these parts will be appropriately omitted. Figure 4 In (b), Figure 3 (b) The same figure.

[0058] The display system 10B includes a wide-angle camera 101 , a normal camera 102 , a GPU 103 , a display 104 , and an eye tracking system 105 .

[0059] The eye tracking system 105 analyzes the facial image of the user (person) captured by, for example, an infrared camera in real time to obtain gaze information. Then, based on the gaze information, the eye tracking system 105 controls the movement of the imaging direction of the ordinary camera 102 so that the imaging direction matches the gaze. Based on the gaze information, the eye tracking system 105 also controls the movement of the focus area (high-resolution area) so that the focus area matches the gaze.

[0060] Figure 4 The other parts of the display system 10B shown in (a) are Figure 3 The display system 10A shown in (a) is configured in the same manner.

[0061] and Figure 3 Like the display system 10A shown in (a), in Figure 4 In the display system 10B shown in (a) of FIG. 1 , both high-resolution real-space images and reduced system load can be achieved without having to capture camera input at 4K resolution.

[0062] exist Figure 4In the display system 10B shown in (a), the movement of the imaging direction of the ordinary camera device 102 is controlled to match the user's gaze, and the movement of the focus area (high-resolution area) within the 4K resolution display image is also controlled, thereby allowing the user to always view the real space image in the gaze direction with high resolution.

[0063] "Third configuration example of display system"

[0064] exist Figure 5 In (a), a configuration example of a display system 10C for real space images as an embodiment is shown, which is installed in a head-mounted display and used in a video see-through AR device, an MR device, etc. The configuration example shown here is only the part for one eye. Figure 5 In (a), Figure 4 The same reference numerals are used for the parts corresponding to the parts in (a), and thus detailed descriptions of these parts will be appropriately omitted.

[0065] The display system 10C includes a wide-angle camera 101 , a normal camera 102 , a GPU 103 , a display 104 , an eye tracking system 105 , a telephoto camera 106 , and a subject distance measurement system 107 .

[0066] The telephoto camera 106 constitutes a telephoto image capturing unit and, like the normal camera 102, has a narrow viewing angle but can capture a narrow range at high resolution. Like the normal camera 102, the telephoto camera 106 also constitutes a zoom camera because it can capture a narrow range at high resolution. Figure 5 As shown in (b), like the normal camera 102, the telephoto camera 106 captures an image at a viewing angle θ2 corresponding to ¼ of the imaging range of the wide-angle camera 101, and outputs an image with a 1K (1080×1080) resolution.

[0067] In this display system 10C, the normal camera 102 or the telephoto camera 106 is selectively used as the magnification camera. Here, the normal camera 102 is suitable for an imaging distance of, for example, 10 cm or more and less than 10 m, while the telephoto camera 106 is suitable for an imaging distance of, for example, 10 m or more.

[0068] The subject distance measurement system 107 acquires information about the distance to the subject in the gaze direction using, for example, a SLAM camera. The acquisition of this distance information is not limited to using a SLAM camera, and may be performed using other methods.

[0069] The subject distance measuring system 107 dynamically switches the camera to be used as the zoom camera to the normal camera 102 or the telephoto camera 106 according to the distance to the subject in the gaze direction of the user. In this case, for example, when the distance to the subject in the gaze direction is less than 10 m, the camera is switched to the normal camera 102, and when the distance to the subject in the gaze direction is 10 m or more, the camera is switched to the telephoto camera 106.

[0070] The eye tracking system 105 analyzes the facial image of the user (person) captured by, for example, an infrared camera in real time to obtain gaze information. Then, based on the gaze information, the eye tracking system 105 controls the movement of the imaging direction of the zoom camera (normal camera 102, telephoto camera 106) so that the imaging direction matches the gaze. Based on the gaze information, the eye tracking system 105 also controls the movement of the focus area (high-resolution area) so that the focus area matches the gaze.

[0071] Figure 5 The other parts of the display system 10C shown in (a) are Figure 4 The display system 10B shown in (a) is configured in the same manner.

[0072] and Figure 3 Like the display system 10A shown in (a), in Figure 5 In the display system 10C shown in (a) of FIG. 1 , both high-resolution real-space images and reduced system load can be achieved without having to capture camera input at 4K resolution.

[0073] and Figure 4 Like the display system 10B shown in (a), in Figure 5 In the display system 10C shown in (a), the movement of the imaging direction of the magnifying camera (normal camera 102, telephoto camera 106) is controlled to match the user's gaze, and the movement of the focus area (high-resolution area) within the 4K resolution display image is also controlled, thereby allowing the user to always view the real space image in the gaze direction with high resolution.

[0074] exist Figure 5 In the display system 10C shown in (a), the camera device to be used as a magnifying camera device can be dynamically switched to a normal camera device 102 or a telephoto camera device 106 according to the distance to the subject in the user's gaze direction, thereby allowing the user to always view the real space image in the gaze direction in an appropriate state (for example, in a more focused state).

[0075] "Fourth configuration example of display system"

[0076] exist Figure 6 In (a), a configuration example of a display system 10D for real space images as an embodiment is shown, which is installed in a head-mounted display and used in a video see-through AR device, an MR device, etc. The configuration example shown here is only the part for one eye. Figure 6 In (a), Figure 5 The same reference numerals are used for the parts corresponding to the parts in (a), and detailed descriptions of these parts will be appropriately omitted.

[0077] The display system 10D includes a wide-angle camera 101 , a normal camera 102 , a GPU 103 , a display 104 , an eye tracking system 105 , a subject distance measuring system 107 , and a close-up camera (macro camera) 108 .

[0078] The close-up camera 108 constitutes a close-up image capturing unit and has a narrow viewing angle like the normal camera 102, but can capture a narrow range with high resolution. Like the normal camera 102, the close-up camera 108 also constitutes a magnified camera because it can capture a narrow range with high resolution. Figure 6 As shown in (b), like the normal camera 102, the close-up camera 108 captures an image at a viewing angle θ2 corresponding to 1 / 4 of the imaging range of the wide-angle camera 101, and outputs an image with a 1K (1080×1080) resolution.

[0079] In this display system 10D, the normal camera 102 or the close-up camera 108 is selectively used as the magnification camera. Here, the normal camera 102 is suitable for an imaging distance of, for example, 10 cm or more and less than 10 m, while the close-up camera 108 is suitable for an imaging distance of, for example, less than 10 cm.

[0080] The subject distance measurement system 107 acquires information about the distance to the subject in the gaze direction using, for example, a SLAM camera. The acquisition of this distance information is not limited to using a SLAM camera, and may be performed using other methods.

[0081] The subject distance measuring system 107 dynamically switches the camera to be used as the zoom camera to the normal camera 102 or the close-up camera 108 according to the distance to the subject in the gaze direction of the user. In this case, for example, when the distance to the subject in the gaze direction is 10 cm or more, the camera is switched to the normal camera 102, and when the distance to the subject in the gaze direction is less than 10 cm, the camera is switched to the close-up camera 108.

[0082] The eye tracking system 105 analyzes the facial image of the user (person) captured by, for example, an infrared camera in real time to obtain gaze information. Then, based on the gaze information, the eye tracking system 105 controls the movement of the imaging direction of the zoom camera (normal camera 102, close-up camera 108) so that the imaging direction matches the gaze. Based on the gaze information, the eye tracking system 105 also controls the movement of the focus area (high-resolution area) so that the focus area matches the gaze.

[0083] Figure 6 The other parts of the display system 10D shown in (a) are Figure 5 The display system 10C shown in (a) is configured in the same manner.

[0084] and Figure 3 Like the display system 10A shown in (a), in Figure 6 In the display system 10D shown in (a) of FIG. 1 , both high-resolution real-space images and reduced system load can be achieved without having to capture camera input at 4K resolution.

[0085] and Figure 4 Like the display system 10B shown in (a), in Figure 6 In the display system 10D shown in (a), the movement of the imaging direction of the magnifying camera (normal camera 102, close-up camera 108) is controlled to match the user's gaze, and the movement of the focus area (high-resolution area) within the 4K resolution display image is also controlled, thereby allowing the user to always view the real space image in the gaze direction with high resolution.

[0086] exist Figure 6 In the display system 10D shown in (a), the camera device to be used as a magnifying camera device can be dynamically switched to a normal camera device 102 or a close-up camera device 108 according to the distance to the subject in the user's gaze direction, thereby allowing the user to always view the real space image in the gaze direction in an appropriate state (for example, in a more focused state).

[0087] "Fifth configuration example of the display system"

[0088] exist Figure 7 In (a), a configuration example of a display system 10E for real space images as an embodiment is shown, which is installed in a head-mounted display and used in a video see-through AR device, an MR device, etc. The configuration example shown here is only the part for one eye. Figure 7 In (a), Figure 5 (a) and Figure 6The same reference numerals are used for the parts corresponding to the parts in (a), and detailed descriptions of these parts will be appropriately omitted.

[0089] The display system 10E includes a wide-angle camera 101 , a normal camera 102 , a GPU 103 , a display 104 , an eye tracking system 105 , a telephoto camera 106 , a subject distance measuring system 107 , and a close-up camera (macro camera) 108 .

[0090] The telephoto camera 106 constitutes a telephoto image capturing unit and has a narrow angle of view like the normal camera 102, but can capture a narrow range with high resolution. The close-up camera 108 constitutes a close-up image capturing unit and has a narrow angle of view like the normal camera 102, but can capture a narrow range with high resolution.

[0091] Like the normal camera 102, the telephoto camera 106 and the close-up camera 108 also constitute zoom cameras because they can each capture a narrow range at high resolution. Figure 7 As shown in (b), like the ordinary camera 102, the telephoto camera 106 and the close-up camera 108 each capture an image at a viewing angle θ2 corresponding to 1 / 4 of the imaging range of the wide-angle camera 101, and output an image with a resolution of 1K (1080x1080).

[0092] In this display system 10E, a normal camera 102, a telephoto camera 106, or a close-up camera 108 is selectively used as a magnifying camera. Here, the normal camera 102 is suitable for an imaging distance of, for example, 10 cm or more and less than 10 m, the telephoto camera 106 is suitable for an imaging distance of, for example, 10 m or more, and the close-up camera 108 is suitable for an imaging distance of, for example, less than 10 cm.

[0093] The subject distance measurement system 107 acquires information about the distance to the subject in the gaze direction using, for example, a SLAM camera. The acquisition of this distance information is not limited to using a SLAM camera, and may be performed using other methods.

[0094] The subject distance measuring system 107 dynamically switches the camera to be used as the zoom camera to the normal camera 102, the telephoto camera 106, or the close-up camera 108, according to the distance to the subject in the gaze direction of the user. In this case, for example, when the distance to the subject in the gaze direction is 10 cm or more and less than 10 m, the camera is switched to the normal camera 102; when the distance to the subject in the gaze direction is 10 m or more, the camera is switched to the telephoto camera 106; and when the distance to the subject in the gaze direction is less than 10 cm, the camera is switched to the close-up camera 108.

[0095] The eye tracking system 105 analyzes the facial image of the user (person) captured by, for example, an infrared camera in real time to obtain gaze information. Then, based on the gaze information, the eye tracking system 105 controls the movement of the imaging direction of the zoom camera (normal camera 102, telephoto camera 106, close-up camera 108) so that the imaging direction matches the gaze. Based on the gaze information, the eye tracking system 105 also controls the movement of the focus area (high-resolution area) so that the focus area matches the gaze.

[0096] Figure 7 The other parts of the display system 10E shown in (a) are Figure 5 The display system 10C shown in (a) and Figure 6 The display system 10D shown in (a) is configured in the same manner.

[0097] and Figure 3 Like the display system 10A shown in (a), in Figure 7 In the display system 10E shown in (a) of FIG. 1 , both high-resolution real-space images and reduced system load can be achieved without having to capture camera input at 4K resolution.

[0098] and Figure 4 Like the display system 10B shown in (a), in Figure 7 In the display system 10E shown in (a), the movement of the imaging direction of the magnifying camera (normal camera 102, telephoto camera 106, close-up camera 108) is controlled to match the user's gaze, and the movement of the focus area (high-resolution area) within the 4K resolution display image is also controlled, thereby allowing the user to always view the real space image in the gaze direction with high resolution.

[0099] exist Figure 7In the display system 10E shown in (a), the camera device to be used as a magnifying camera device can be dynamically switched to a normal camera device 102, a telephoto camera device 106 or a close-up camera device 108 according to the distance to the subject in the user's gaze direction, thereby allowing the user to always view the real space image in the gaze direction in an appropriate state (for example, in a more focused state).

[0100] "Displays a configuration diagram of data transfer on the system"

[0101] Figure 8 It is shown Figure 7 0 is a configuration diagram showing an example of data transmission in a display system 10E.

[0102] “SLAM camera right” refers to a SLAM camera on the right side for acquiring information about the distance to a subject in a gaze direction, and “SLAM camera left” refers to a SLAM camera on the left side for acquiring information about the distance to a subject in a gaze direction. “IR camera right” refers to an infrared (IR) camera on the right side for acquiring gaze information of a user, and “IR camera left” refers to an IR camera on the left side for acquiring gaze information of a user.

[0103] “Wide-angle camera right” refers to the wide-angle camera on the right side, and “wide-angle camera left” refers to the wide-angle camera on the left side. “Normal camera right” refers to the normal camera on the right side, and “Normal camera left” refers to the normal camera on the left side. “Telephoto camera right” refers to the telephoto camera on the right side, and “Telephoto camera left” refers to the telephoto camera on the left side. “Macro camera right” refers to the close-up camera (macro camera) on the right side, and “Macro camera left” refers to the close-up camera (macro camera) on the left side.

[0104] The "memory" refers to a memory that temporarily stores the output data of each camera. In this case, the output data from each camera is transmitted to the "memory" via the "CSI" (ie, camera serial interface).

[0105] The "camera driver" refers to a driver that controls each camera. The "camera middleware" imports output data from the IR camera and analyzes the facial image to obtain gaze information. The "camera middleware" also imports and processes output data from the SLAM camera to obtain information about the distance to the subject in the gaze direction.

[0106] The "application" controls the movement of the imaging direction of the magnifying camera (normal camera, telephoto camera, close-up camera) based on the gaze information obtained by the "camera middleware" so that the imaging direction matches the gaze. The "application" also dynamically switches the camera to be used as the magnifying camera to the normal camera, telephoto camera, or close-up camera based on the information about the distance to the subject in the gaze direction obtained by the "camera middleware".

[0107] The "application" also sets the position of the focus area (high-resolution area) to match the gaze based on the gaze information obtained by the "camera middleware". The "rendering middleware" refers to the module that performs synthesis, rendering processing, and display processing. The "GPU driver" refers to the GPU control driver. The "GPU memory" refers to the memory used by the "rendering middleware" to perform synthesis and rendering processing to generate display images.

[0108] In this synthesis and rendering process, in order to generate a 4K resolution display image, a narrow range 1K resolution image captured at high resolution using an enlarged camera is directly used as an image of the focus area (high resolution area), and a 4K resolution image obtained by enlarging a wide range 1K resolution image captured using a wide-angle camera is used as an image of the peripheral area (low resolution area). At this time, the position of the focus area (high resolution area) is set to the position set by the "application" based on the gaze information.

[0109] The "display driver" configures the display panel. The "display" refers to the hardware module that transfers the display image in the "GPU memory" to the panel. The "panel right" refers to the display panel on the right, and the "panel left" refers to the display panel on the left. The right display image generated by the GPU performing synthesis and rendering processing is transmitted from the "GPU memory" via the "display" and then via the "DSI" (i.e., display serial interface) to the "panel right", and the display image is displayed at the "panel right". Similarly, the left display image generated by the GPU performing synthesis and rendering processing is transmitted from the "GPU memory" via the "display" and then via the "DSI" (i.e., display serial interface) to the "panel left", and the display image is displayed at the "panel left".

[0110] "Configuration example of information processing device"

[0111] Fig. 9 Shown with Figure 7The configuration example (functional configuration example) of the information processing device 200 corresponding to the display system 10E shown in (a) of FIG. The information processing device 200 includes: a wide-angle image capturing unit 201, a normal image capturing unit 202, a telephoto image capturing unit 203, a close-up image capturing unit 204, an image synthesis unit 205, a display unit 206, a subject distance measuring unit 207, an image capturing unit switching control unit 208, a gaze detecting unit 209, and an imaging direction / image synthesis control unit 210.

[0112] The wide-angle image capturing unit 201 corresponds to the wide-angle camera 101, the normal image capturing unit 202 corresponds to the normal camera 102, the telephoto image capturing unit 203 corresponds to the telephoto camera 106, and the close-up image capturing unit 204 corresponds to the close-up camera 108. The normal image capturing unit 202, the telephoto image capturing unit 203, and the close-up image capturing unit 204 constitute a magnified image capturing unit.

[0113] The gaze detection unit 209 and the imaging direction / image synthesis control unit 210 correspond to the eye tracking system 105. The subject distance measurement unit 207 and the image capturing unit switching control unit 208 correspond to the subject distance measurement system 107.

[0114] The gaze detection unit 209 analyzes in real time a facial image of a user (person) captured by, for example, an infrared camera to obtain gaze information. The subject distance measurement unit 207 obtains information about the distance to the subject in the gaze direction of the user based on, for example, output data from the SLAM camera and further gaze information obtained by the gaze detection unit 209.

[0115] The image capture unit switching control unit 208 dynamically switches the image capture unit to be used as an enlarged image capture unit to a normal image capture unit 202, a telephoto image capture unit 203, or a close-up image capture unit 204 according to the distance to the subject in the user's gaze direction acquired by the subject distance measurement unit 207.

[0116] The imaging direction / image synthesis control unit 210 controls the movement of the imaging direction of the enlarged image capture unit (normal image capture unit 202, telephoto image capture unit 203, close-up image capture unit 204) based on the gaze information acquired by the gaze detection unit 209 so that the imaging direction matches the gaze.

[0117] The image synthesis unit 205 corresponds to the GPU 103. The image generation unit 205 synthesizes an image based on a focus area (high-resolution area) of an image captured by the enlarged image capture unit (normal image capture unit 202, telephoto image capture unit 203, close-up image capture unit 204) with an image based on a peripheral area (low-resolution area) around the focus area of ​​an image captured by the wide-angle image capture unit 201 (i.e., performs foveated rendering) to generate a 4K resolution display image.

[0118] Here, as the image of the focus area (high-resolution area), the 1K resolution image captured by the enlarged image capture unit (normal image capture unit 202, telephoto image capture unit 203, close-up image capture unit 204) is used directly. As the image of the peripheral area, a 4K resolution image obtained by enlarging the 1K resolution image captured by the wide-angle image capture unit 201 by 4 times is used.

[0119] In this case, the imaging direction / image synthesis control unit 210 controls the movement of the position of the focus area (high-resolution area) based on the gaze information acquired by the gaze detection unit 209 so that the position matches the gaze.

[0120] The display unit 206 corresponds to the display 104 . The display unit 206 displays the display image generated by the image synthesis unit 205 .

[0121] Fig.10 The flowchart shows Fig. 9 . First, in step ST1, the information processing device 200 starts processing. Next, in step ST2, the information processing device 200 determines whether the user has performed an operation to start the video perspective mode. When the user has performed an operation to start the video perspective mode, in step ST3, the information processing device 200 starts the operation of the image capture unit (wide-angle image capture unit 201, normal image capture unit 202, telephoto image capture unit 203, and close-up image capture unit 204).

[0122] Next, in step ST4, the information processing device 200 sets the normal image capturing unit 202 to be used as the enlarged image capturing unit through the image capturing unit switching control unit 208. Next, in step ST5, assuming that the user's gaze is directed to the center of the screen, the information processing device 200 performs initial settings of the imaging direction and the focus area (high-resolution area) of the enlarged image capturing unit through the imaging direction / image synthesis unit 210.

[0123] Next, in step ST6, in order to generate a 4K resolution display image, the information processing device 200 allocates the enlarged image (1K resolution image captured by the enlarged image capturing unit) to the focus area (high resolution area) through the image synthesis unit 205, and enlarges the wide-angle image (1K resolution image captured by the wide-angle image capturing unit 201) by 4 times, and allocates the resulting image to the peripheral area. Next, in step ST7, the information processing device 200 displays the display image as a real space image on the display unit 206.

[0124] Next, in step ST8, the information processing device 200 determines whether the gaze of the user acquired by the gaze detection unit 209 has changed. In determining whether the gaze has changed, when the gaze has changed by a preset threshold value or more, it can be determined that the gaze has changed. This makes it possible to reduce the frequency of changing the imaging direction and the focus area (high-resolution area) of the enlarged image capture unit, thereby reducing the processing load. When the user's gaze has not changed, the information processing device 200 returns to the processing of step ST6 to repeat the same processing as described above.

[0125] On the other hand, when the user's gaze has changed, then in step ST9, the information processing device 200 controls (changes) the movement of the imaging direction of the enlarged image capture unit through the imaging direction / image synthesis unit 210 so that the imaging direction matches the gaze. Next, in step ST10, the information processing device 200 controls (changes) the position of the focus area (high-resolution area) through the imaging direction / image synthesis unit 210 so that it matches the gaze. After the processing of step ST10, the information processing device 200 proceeds to the processing of step ST11.

[0126] In step ST11, the information processing device 200 determines whether the distance to the subject in the user's gaze direction obtained by the subject distance measurement unit 207 is a medium distance (e.g., 10 cm or greater and less than 10 m), a long distance (e.g., 10 m or greater), or a short distance (e.g., less than 10 cm).

[0127] When the distance to the subject in the user's gaze direction is a medium distance, in step ST12, the information processing device 200 sets the normal image capture unit 202 to be used as the enlarged image capture unit through the image capture unit switching control unit 208. When the distance to the subject in the user's gaze direction is a long distance, in step ST13, the information processing device 200 sets the telephoto image capture unit 203 to be used as the enlarged image capture unit through the image capture unit switching control unit 208. When the distance to the subject in the user's gaze direction is a short distance, in step ST14, the information processing device 200 sets the close-up image capture unit 204 to be used as the enlarged image capture unit through the image capture unit switching control unit 208.

[0128] After the processing of steps ST12, ST13, and ST14, the information processing device 200 determines in step ST15 whether the user has performed an operation to end the video perspective mode. When the user has not performed an operation to end the video perspective mode, the information processing device 200 returns to the processing of step ST6 to repeat the same processing as described above.

[0129] On the other hand, when the user has performed an operation to end the video perspective mode, then in step ST16, the information processing device 200 ends the operation of the image capture unit (wide-angle image capture unit 201, normal image capture unit 202, telephoto image capture unit 203, and close-up image capture unit 204), and then returns to the processing of step ST2.

[0130] exist Fig. 9 In the information processing device 200 shown in , both a high-resolution real-world image and a reduced system load can be achieved without capturing an image capture unit input (camera input) at 4K resolution.

[0131] exist Fig. 9 In the information processing device 200 shown in , the movement of the imaging direction of the magnified image capture unit (normal image capture unit 202, telephoto image capture unit 203, close-up image capture unit 204) is controlled to match the user's gaze, and the movement of the focus area (high-resolution area) within the 4K resolution display image is also controlled, thereby allowing the user to always view the real space image in the gaze direction with high resolution.

[0132] exist Fig. 9In the information processing device 200 shown in , the image capture unit to be used as a magnified image capture unit can be dynamically switched to a normal image capture unit 202, a telephoto image capture unit 203, or a close-up image capture unit 204 according to the distance to the subject in the user's gaze direction, thereby allowing the user to always view the real space image in the gaze direction in an appropriate state (for example, in a more focused state).

[0133] As mentioned above, Fig. 9 The information processing device 200 shown in FIG. Figure 7 Although detailed description will be omitted, it goes without saying that Figures 3 to 6 The information processing devices corresponding to the display systems 10A to 10D shown in FIG. 1 can be configured in a similar manner.

[0134] <2. Modification example>

[0135] Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to these examples. It should be obvious to those skilled in the art of the present disclosure that various changed or modified examples can be made within the scope of the technical spirit described in the claims, and should certainly be interpreted as falling within the technical scope of the present disclosure.

[0136] In addition, the effects described herein are merely illustrative or exemplary and are not intended to be limiting. Therefore, in addition to or in place of the above effects, the technology according to the present disclosure may exhibit other effects that are obvious to those skilled in the art based on the description herein.

[0137] The present technology can be configured as follows.

[0138] (1) An information processing device comprising:

[0139] an image acquisition unit that acquires a first image captured at a first angle of view and a second image captured at a second angle of view within the first angle of view that is narrower than the first angle of view; and

[0140] An image generating unit generates a display image by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image and surrounding the high-resolution area.

[0141] (2) The information processing device according to (1), wherein:

[0142] The image of the high-resolution area is an image directly used as the second image, and

[0143] The image of the peripheral area is an image obtained by enlarging the first image.

[0144] (3) The information processing device according to (2), wherein:

[0145] The first image and the second image each have a first resolution, and

[0146] The image of the peripheral area is an image having a second resolution obtained by enlarging the first image according to a ratio between the first viewing angle and the second viewing angle.

[0147] (4) The information processing device according to (3), wherein:

[0148] The first resolution is 1K resolution, and

[0149] The second resolution is 4K resolution.

[0150] (5) The information processing device according to any one of (1) to (4) further includes a control unit, which controls the movement of the imaging direction of the image capture unit used to obtain the second image and controls the movement of the position of the high-resolution area based on the user's gaze information.

[0151] (6) The information processing device according to (5), further including a gaze detection unit that detects gaze information of the user.

[0152] (7) The information processing device according to any one of (1) to (6), further including a control unit that controls switching of an image capturing unit for obtaining the second image based on information about a distance to a subject related to the second image.

[0153] (8) The information processing device according to (7), further including a subject distance measurement unit configured to obtain information on the distance to the subject related to the second image.

[0154] (9) An information processing device according to (7) or (8), wherein the control unit switches the image capture unit used to obtain the second image to a first image capture unit for a first imaging distance or a second image capture unit for an imaging distance longer than or shorter than the first imaging distance.

[0155] (10) The information processing device according to (9), wherein:

[0156] The first image capturing unit is a common image capturing unit, and

[0157] The second image capturing unit is a telephoto image capturing unit or a close-up image capturing unit.

[0158] (11) An information processing device according to (7) or (8), wherein the control unit switches the image capture unit used to obtain the second image to any one of: a first image capture unit for a first imaging distance, a second image capture unit for an imaging distance longer than the first imaging distance, and a third image capture unit for an imaging distance shorter than the first imaging distance.

[0159] (12) The information processing device according to (11), wherein:

[0160] The first image capturing unit is a common image capturing unit,

[0161] The second image capture unit is a telephoto image capture unit, and

[0162] The third image capturing unit is a close-up image capturing unit.

[0163] (13) The information processing device according to any one of (1) to (12), further comprising:

[0164] a wide-angle image capturing unit, the wide-angle image capturing unit being used to obtain the first image; and

[0165] A magnified image capturing unit is used to obtain the second image.

[0166] (14) The information processing device according to any one of (1) to (13), further including a display unit that displays the display image.

[0167] (15) An information processing method comprising the following steps:

[0168] Acquire a first image captured at a first viewing angle and a second image captured at a second viewing angle within the first viewing angle and narrower than the first viewing angle; and

[0169] A display image is generated by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image of a peripheral area around the high-resolution area.

[0170] (16) A program for causing a computer to execute an information processing method, the information processing method comprising the following steps:

[0171] Acquire a first image captured at a first viewing angle and a second image captured at a second viewing angle within the first viewing angle and narrower than the first viewing angle; and

[0172] A display image is generated by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image of a peripheral area around the high-resolution area.

[0173] [reference numerals list]

[0174] 10A to 10E Display Systems

[0175] 101 Wide-angle camera device

[0176] 102 General camera device

[0177] 103GPU

[0178] 104 Display

[0179] 105 Eye Tracking System

[0180] 106 Telephoto camera device

[0181] 107 Subject Distance Measurement System

[0182] 108 Close-up camera device (macro camera device)

[0183] 200 Information processing device

[0184] 201 Wide-angle image capture unit

[0185] 202 Normal image capture unit

[0186] 203 Telephoto image capture unit

[0187] 204 Close-up image capture unit

[0188] 205 Image Synthesis Unit

[0189] 206 display units

[0190] 207 Subject distance measurement unit

[0191] 208 Image capture unit switching control unit

[0192] 209 Gaze Detection Unit

[0193] 210 Imaging direction / image synthesis control unit

Claims

1. An information processing device, comprising: an image acquisition unit configured to acquire a first image captured at a first viewing angle and a second image captured at a second viewing angle within the first viewing angle and narrower than the first viewing angle; as well as An image generating unit generates a display image by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image and surrounding the high-resolution area.

2. The information processing device according to claim 1, wherein: The image of the high-resolution area is an image directly used as the second image, and The image of the peripheral area is an image obtained by enlarging the first image.

3. The information processing device according to claim 2, wherein: The first image and the second image each have a first resolution, and The image of the peripheral area is an image having a second resolution obtained by enlarging the first image according to a ratio between the first viewing angle and the second viewing angle.

4. The information processing device according to claim 3, wherein: The first resolution is 1K resolution, and The second resolution is 4K resolution.

5. The information processing device according to claim 1, further comprising a control unit that controls movement of an imaging direction of an image capture unit for obtaining the second image and controls movement of a position of the high-resolution area based on gaze information of a user. 6 . The information processing device according to claim 5 , further comprising a gaze detection unit that detects gaze information of the user. 7 . The information processing apparatus according to claim 1 , further comprising a control unit that controls switching of an image capturing unit for obtaining the second image based on information about a distance to a subject related to the second image. 8 . The information processing apparatus according to claim 7 , further comprising a subject distance measurement unit configured to obtain information on the distance to the subject related to the second image.

9. The information processing device according to claim 7, wherein: The control unit switches the image capturing unit for obtaining the second image to a first image capturing unit for a first imaging distance or a second image capturing unit for an imaging distance longer or shorter than the first imaging distance.

10. The information processing device according to claim 9, wherein: The first image capturing unit is a common image capturing unit, and The second image capturing unit is a telephoto image capturing unit or a close-up image capturing unit.

11. The information processing device according to claim 7, wherein: The control unit switches the image capture unit used to obtain the second image to any one of: a first image capture unit for a first imaging distance, a second image capture unit for an imaging distance longer than the first imaging distance, and a third image capture unit for an imaging distance shorter than the first imaging distance.

12. The information processing device according to claim 11, wherein: The first image capturing unit is a common image capturing unit, The second image capture unit is a telephoto image capture unit, and The third image capturing unit is a close-up image capturing unit.

13. The information processing device according to claim 1, further comprising: a wide-angle image capturing unit, the wide-angle image capturing unit being used to obtain the first image; as well as A magnified image capturing unit is used to obtain the second image. 14 . The information processing apparatus according to claim 1 , further comprising a display unit that displays the display image.

15. An information processing method comprising the following steps: Acquire a first image captured at a first viewing angle and a second image captured at a second viewing angle within the first viewing angle and narrower than the first viewing angle; as well as A display image is generated by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image of a peripheral area around the high-resolution area.

16. A program for causing a computer to execute an information processing method, the information processing method comprising the following steps: Acquire a first image captured at a first viewing angle and a second image captured at a second viewing angle within the first viewing angle and narrower than the first viewing angle; as well as A display image is generated by synthesizing an image based on the high-resolution area of ​​the second image and an image based on the first image of a peripheral area around the high-resolution area.

Images