Image display system, image display method, and program

Abstract

To provide an image display system, image display method, and program capable of further suppressing VR sickness.SOLUTION: An image display system includes: an attention point detection part which detects an attention point in which a user viewing an image displayed on a display device is paying attention on the image; a speed parameter setting part which sets a speed parameter according to distance to a subject at the detected attention point; and a display control part which enlarges or reduces the display of the image at a speed corresponding to the set speed parameter.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an image display system, an image display method, and a program. [Background technology]

[0002] Conventionally, virtual reality (VR) can be experienced using a head mounted display (HMD). When checking the details of a VR image using a head mounted display, a symptom called VR sickness (sensory mismatch) may occur due to the difference between one's physical sense and the amount of visual movement when the VR image is enlarged. Various technologies have been proposed to suppress this VR sickness.

[0003] For example, Patent Document 1 below discloses a technology for scaling a display image based on the movement of the user's head. This technology detects the movement of the user's head to determine a scaling factor, and prevents a large deviation between the movement of the head and the center of scaling, thereby suppressing VR sickness. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6339887 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 does not take into consideration the speed at which the display image is enlarged or reduced. Therefore, the display image is enlarged or reduced at a constant speed regardless of the amount of head movement and the scaling factor. This causes a discrepancy between the physical sensation and the amount of visual movement for the user, which may cause VR sickness.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide an image display system, an image display method, and a program capable of further suppressing VR sickness. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, one embodiment of the image display system of the present invention is an image display system comprising: an attention point detection unit that detects an attention point in an image that is being focused on by a user viewing an image displayed on a display device; a speed parameter setting unit that sets a speed parameter according to the distance to a subject at the detected attention point; and a display control unit that enlarges or reduces the display of the image at a speed according to the set speed parameter.

[0008] An image display method according to one embodiment of the present invention is an image display method executed by a computer, including a focus detection process for detecting a focus in an image displayed on a display device by a user viewing the image, a speed parameter setting process for setting a speed parameter according to a distance to a subject at the detected focus, and a display control process for enlarging or reducing a display of the image at a speed according to the set speed parameter.

[0009] A program according to one aspect of the present invention is a program for causing a computer to function as: an attention point detection means for detecting an attention point in an image displayed on a display device at which a user viewing the image is focusing; a speed parameter setting means for setting a speed parameter according to the distance to a subject at the detected attention point; and a display control means for enlarging or reducing the display of the image at a speed according to the set speed parameter. Effect of the Invention

[0010] According to the present invention, VR sickness can be further suppressed. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of a panoramic VR system according to an embodiment of the present invention. [Diagram 2] 11A to 11C are diagrams illustrating an example of control of the shooting angle in the pan direction and the tilt direction according to the present embodiment. [Diagram 3] 11A and 11B are diagrams illustrating an example of control of the focal center position in the horizontal direction according to the embodiment. [Figure 4] 2 is a block diagram showing an example of a functional configuration of an imaging control device according to the present embodiment. FIG. [Diagram 5] FIG. 2 is a diagram illustrating an example of an omnidirectional image according to the embodiment. [Figure 6] FIG. 11 is a diagram showing an example of imaging area determination according to the present embodiment. [Figure 7] 1 is a block diagram showing an example of a functional configuration of an image editing apparatus according to an embodiment of the present invention; [Figure 8] 2 is a block diagram showing an example of a functional configuration of a display control device according to the present embodiment. FIG. [Figure 9] 13 is a diagram showing an example of a speed parameter when a point of interest is in a long distance according to the present embodiment; FIG. [Figure 10] 13 is a diagram showing an example of a speed parameter when a point of interest according to the present embodiment is in a short distance; [Figure 11] 11A and 11B are diagrams illustrating an example of movement of a three-dimensional object when enlarging a display of a panoramic image according to the present embodiment. [Figure 12] 11A and 11B are diagrams illustrating an example of movement of a three-dimensional object when the display of a panoramic image according to the present embodiment is reduced. [Figure 13] 10 is a flowchart showing an example of a processing flow in pre-imaging according to the present embodiment. [Figure 14] 10 is a flowchart showing an example of a processing flow in actual shooting according to the present embodiment. [Figure 15] 10 is a flowchart showing an example of a processing flow in image editing according to the embodiment. [Figure 16]5 is a flowchart showing an example of a processing flow for image viewing according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, an X-axis, a Y-axis, or a Z-axis that are mutually orthogonal as necessary is shown. On each axis, the direction in which the arrow extends is called the "positive direction," and the direction opposite to the positive direction is called the "negative direction." In the following, a system (hereinafter, also referred to as a "panoramic VR system") that allows a panoramic image generated based on an image captured in a real space to be viewed in virtual reality (VR) will be described. As an example, a user views a panoramic image displayed in a VR space using a head mounted display (HMD).

[0013] <1. Panoramic VR system configuration> The configuration of a panoramic VR system according to this embodiment will be described with reference to Fig. 1 to Fig. 3. Fig. 1 is a block diagram showing an example of the configuration of a panoramic VR system according to this embodiment. 1, the panoramic VR system 1000 includes an imaging system 1, an image editing system 2, and an image display system 3. The network NW may include, for example, a local area network (LAN), a wide area network (WAN), a telephone network (mobile phone network, landline telephone network, etc.), a regional Internet Protocol (IP) network, the Internet, etc., as a configuration for transmitting and receiving information.

[0014] (1) Shooting System 1 The photography system 1 is a system for photographing a space (hereinafter also referred to as a "target space") that is a target for generating a panoramic image to be displayed in a VR space. Images (hereinafter also referred to as "photographed images") photographed by the photography system 1 of each photography area in the target space are used to generate a panoramic image to be displayed in VR. As shown in FIG. 1, the photography system 1 includes a photography control device 10, an omnidirectional camera 11, a depth camera 12, a photography camera 13, and a camera platform 14.

[0015] (1-1) Imaging control device 10 The photography control device 10 is a device for controlling photography of a target space. The photography control device 10 is, for example, a device such as a PC (Personal Computer), a server device, a smartphone, or a tablet terminal. In the panoramic VR system 1000, the shooting control device 10 is communicably connected to the image editing device 20 of the image editing system 2 via the network NW. In the shooting system 1, the shooting control device 10 is communicably connected to the omnidirectional camera 11, the depth camera 12, the shooting camera 13, and the camera platform 14 by wired or wireless connection.

[0016] (1-2) Omnidirectional Camera 11 The omnidirectional camera 11 is a camera that captures images in all directions (360 degrees) of a target space. The omnidirectional camera 11 is also called a 360-degree camera or a panoramic camera. The omnidirectional camera 11 captures images in all directions of a target space from a capture position and transmits the captured images (hereinafter also called "omnidirectional images") to the capture control device 10. The omnidirectional camera 11 captures images while being mounted on, for example, a camera platform 14. In this case, the shooting position of the omnidirectional camera 11 is the position where the camera platform 14 is provided. The omnidirectional camera 11 is mounted on the camera platform 14 and captures omnidirectional images under the control of the shooting control device 10.

[0017] Although the omnidirectional image captured by the omnidirectional camera 11 is a panoramic image, it is not used as a panoramic image to be displayed in the VR space, but is an image captured in advance as an image for preparing the panoramic image.

[0018] (1-3) Depth Camera 12 The depth camera 12 is a camera for acquiring depth information in a target space. The depth camera 12 measures depth information (e.g., distance to a subject) in all directions in the target space from a shooting position, and transmits the measured depth information to the shooting control device 10. The depth camera 12 performs imaging, for example, while being mounted on a camera platform 14. In this case, the imaging position of the depth camera 12 is the position where the camera platform 14 is mounted. The depth camera 12 is mounted on the camera platform 14 and measures depth information based on the control of the imaging control device 10.

[0019] (1-4) Camera 13 The photographing camera 13 is a camera for photographing images, and is an example of a photographing device. The photographing camera 13 photographs the target space for each photographing area from a photographing position, and transmits the photographed photographed images to the photographing control device 10. The photographing camera 13 photographs, for example, while being mounted on a pan head 14. In this case, the photographing position of the photographing camera 13 is the position where the pan head 14 is mounted. The photographing camera 13 is mounted on the pan head 14 and photographs the photographed images based on the control by the photographing control device 10.

[0020] The photographing camera 13 has a lens whose focal length can be electrically controlled. The focal length of the lens is controlled by the photographing control device 10 when photographing an image, for example.

[0021] (1-5) Panhead 14 The camera platform 14 is a platform on which a camera is fixed and used for controlling shooting by remote operation from a remote location. The pan and tilt of the camera platform 14 can be electrically controlled, and the shooting angle of the fixed camera can be adjusted. The shooting angle is adjusted by the shooting control device 10. The shooting control device 10 can adjust the shooting angle in the horizontal (left and right) direction by controlling the pan, and can adjust the shooting angle in the vertical (up and down) direction by controlling the tilt, for example.

[0022] Here, control of the shooting angle in the pan direction and the tilt direction will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of control of the shooting angle in the pan direction and the tilt direction according to this embodiment. Fig. 2 shows an example in which a shooting camera 13 is fixed to a pan head 14.

[0023] As shown in FIG. 2, the camera platform 14 has a rotation axis A1 and a rotation axis A2. The photographing camera 13 is fixed to the rotation axis A2. The photographing control device 10 rotates the base A3 by rotating the rotation axis A1, and can rotate the photographing camera 13 in the direction indicated by the arrow B1 (pan direction). This allows the photographing control device 10 to adjust the photographing angle in the pan direction of the photographing camera 13. In addition, the photographing control device 10 rotates the rotation axis A2 to rotate the photographing camera 13 in the direction indicated by the arrow B2 (tilt direction). This allows the photographing control device 10 to adjust the photographing angle in the tilt direction of the photographing camera 13.

[0024] The camera platform 14 can electrically control the horizontal movement of the fixed camera, and can adjust the shooting position of the camera. The shooting position is adjusted by the shooting control device 10. For example, the shooting control device 10 can adjust the focal center position of the lens to the no-parallax point by adjusting the shooting position. The no-parallax point is the rotation center position in the pan direction when adjusting the camera's shooting angle. The shooting control device 10 can suppress the occurrence of misalignment (parallax) between captured images that occurs when multiple captured images are stitched together by aligning the focal center position with the rotation center position.

[0025] Here, control of the focal center position in the horizontal direction will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of control of the focal center position in the horizontal direction according to this embodiment. Fig. 3 shows, as an example, an example in which the photographing camera 13 is fixed to a pan head 14. Fig. 3 is a diagram of the photographing camera 13 fixed to the pan head 14 in Fig. 2 as viewed from above (positive direction of the Z axis).

[0026] 3, the base A3 of the camera platform 14 has a slide mechanism. The photography control device 10 can move the photographing camera 13 in the direction indicated by the arrow B3-1 or the direction indicated by the arrow B3-2 by sliding the base A3. This allows the photography control device 10 to align the focal center position P1 of the photographing camera 13 with the rotation center position P2 (the center position of the rotation axis A1).

[0027] (2) Image editing system 2 The image editing system 2 is a system for editing the images captured by the image capturing system 1. As shown in FIG. 1, the image editing system 2 includes an image editing device 20.

[0028] (2-1) Image editing device 20 The image editing device 20 is a device for editing a captured image. The image editing device 20 is, for example, a PC, a server device, a smartphone, a tablet terminal, or the like. In the panoramic VR system 1000, an image editing device 20 is communicably connected to an imaging control device 10 of an imaging system 1 and a display control device 30 of an image display system 3 via a network NW.

[0029] (3) Image display system 3 The image display system 3 is a system for displaying a panoramic image in a VR space based on a captured image edited by the image editing device 20. For example, the internal space of a three-dimensional object is used for the VR space. The shape of the three-dimensional object is not particularly limited, but will be described as a sphere in the following embodiment. In this case, the panoramic image is attached to a surface on the inside of the spherical three-dimensional object. The image display system 3 generates a three-dimensional object with the panoramic image attached to the inner surface, and displays the internal space of the three-dimensional object so that the user can view it. As shown in FIG. 1, the image display system 3 includes a display control device 30 and an HMD 31.

[0030] (3-1) Display control device 30 The display control device 30 is a device for generating a three-dimensional object and controlling the display of a panoramic image inside the three-dimensional object. The display control device 30 is, for example, a device such as a PC, a server device, a smartphone, or a tablet terminal. In the panoramic VR system 1000, the display control device 30 is communicatively connected to the image editing device 20 of the image editing system 2 via the network NW. In the image display system 3, the display control device 30 is communicatively connected to the HMD 31 by wired or wireless connection.

[0031] (3-2) HMD31 The HMD 31 is a device that displays a panoramic image in a VR space, and is an example of a display device. The HMD 31 displays a three-dimensional object generated by the display control device 30 in the VR space based on the control by the display control device 30. At this time, the HMD 31 displays the panoramic image so that the user feels as if he or she is viewing the panoramic image inside the three-dimensional object.

[0032] The HMD 31 may include a mechanism for estimating a self-position, a mechanism for detecting a user's line of sight, etc. The self-position estimation may be performed by either an outside-in type estimation or an inside-out type estimation. In the case of outside-in type estimation, the HMD 31 estimates its own position using, for example, infrared rays. In this case, the HMD 31 senses light from an externally installed infrared emitting device using a plurality of infrared sensors provided in the HMD 31, and estimates its own position by combining the infrared sensing timing, the infrared incidence angle, IMU (Inertial Measurement Unit) data sensed by an acceleration sensor, etc. Note that the method by which the HMD 31 estimates its own position in the outside-in type is not limited to the method using infrared rays. In the case of inside-out type estimation, the HMD 31 estimates its own position using, for example, a camera. In this case, the HMD 31 observes the surrounding environment with multiple cameras provided in the HMD 31, performs feature point matching from the observed environment, and estimates its own position by performing triangulation from the feature points and data complementation by an IMU. The user's line of sight is detected, for example, by eye tracking using a camera provided in the HMD 31. Note that the method by which the HMD 31 estimates its own position in the inside-out type is not limited to the method using a camera.

[0033] <2. Functional configuration of the imaging control device> The configuration of the panoramic VR system 1000 according to this embodiment has been described above. Next, the functional configuration of the shooting control device 10 according to this embodiment will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a block diagram showing an example of the functional configuration of the shooting control device 10 according to this embodiment. As shown in FIG. 4, the imaging control device 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0034] (1) Communications Department 110 The communication unit 110 has a function of transmitting and receiving various information. In communication with the omnidirectional camera 11, the communication unit 110 receives an omnidirectional image. In communication with the depth camera 12, the communication unit 110 receives depth information. In communication with the photographing camera 13, the communication unit 110 transmits a control signal for controlling the focal length and photographing (hereinafter also referred to as a "photography control signal") and receives a photographed image. In communication with the tripod head 14, the communication unit 110 transmits a control signal for controlling the operation of the tripod head 14 (hereinafter also referred to as a "tripod head control signal"). In communication with the image editing device 20, the communication unit 110 transmits a photographed image and depth information.

[0035] (2) Input unit 120 The input unit 120 has a function of receiving input. The input unit 120 is, for example, an input device that the imaging control device 10 has as hardware, such as a button, a touch panel, a microphone, a mouse, a keyboard, and the like.

[0036] (3) Storage section 130 The storage unit 130 has a function of storing various information. The function of the storage unit 130 is configured by a storage medium provided as hardware in the imaging control device 10, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. The storage unit 130 stores, for example, an omnidirectional image, information amount information, depth information, shooting information, shot images, etc. The shot images include, for example, high-resolution images, low-resolution images, and focus bracketed images.

[0037] The information amount information is information indicating the amount of information in the target space. For example, the information amount information is information indicating whether the shooting area is an area with a large amount of information or an area with a small amount of information. Note that the information amount information may be information indicating the amount of information possessed by each pixel, a comparison result of the amount of information between pixels, or the like.

[0038] The shooting information is information indicating the classification of the shooting area, the shooting settings, the shooting method, etc. The shooting information is set in advance for each shooting area based on the amount of information each of the multiple shooting areas has in the target space. The classification of the shooting area is information indicating whether the shooting area is a shooting area (first shooting area) to be shot in high definition (high resolution) or a shooting area (second shooting area) to be shot in low definition (low resolution). The shooting settings are information indicating the focal length and shooting angle of the shooting camera 13. The shooting method is information indicating whether or not focus bracket shooting is to be performed.

[0039] The high-resolution image is an image captured by the photographing camera 13 of a high-definition photographing area (hereinafter also referred to as a “high-definition photographing area”) among the photographing images of each of a plurality of photographing areas. The low-resolution image is an image captured by the photographing camera 13 of a low-resolution photographing area (hereinafter also referred to as a "low-resolution photographing area") among the images captured by the photographing camera 13 of each of a plurality of photographing areas. A focus bracketed image is an image obtained by focus bracketing photography, in which the entire area of ​​the corresponding photography area is in focus. In focus bracketing photography, multiple images with different focuses are taken for one photography area, and an image synthesized based on the multiple images is obtained as a focus bracketed image. In this embodiment, of the high-resolution image and the low-resolution image, the image of the photography area in which focus bracketing photography is performed by the photography camera 13 is the focus bracketed image.

[0040] (4) Control unit 140 The control unit 140 has a function of controlling the overall operation of the photography control device 10. The function of the control unit 140 is realized, for example, by causing a central processing unit (CPU) or a graphics processing unit (GPU) provided as hardware in the photography control device 10 to execute a program. As shown in FIG. 4, the control unit 140 includes an information amount information acquisition unit 141, a depth information acquisition unit 142, a shooting area determination unit 143, a shooting setting determination unit 144, a shooting method determination unit 145, a shooting information generation unit 146, and a shooting control unit 147.

[0041] (4-1) Information amount information acquisition unit 141 The information amount information acquisition unit 141 has a function of acquiring information indicating the amount of information in the target space (hereinafter, also referred to as "information amount information"). The information amount information acquisition unit 141 acquires the information amount information of the target space based on the omnidirectional image received by the communication unit 110 from the omnidirectional camera 11. For example, the information amount information acquisition unit 141 determines that an area in the omnidirectional image where the amount of color change between adjacent pixels is large or an area where the frequency is high is an area with a large amount of information, and determines that an area in the omnidirectional image where the amount of color change between adjacent pixels is small or an area where the frequency is low is an area with a small amount of information. After making a judgment on the entire omnidirectional image, the information amount information acquisition unit 141 acquires (generates) information indicating an area with a large amount of information or an area with a small amount of information based on the judgment result as information amount information. The information amount information acquisition unit 141 stores the acquired information amount information in the storage unit 130 , and outputs it to the shooting area determination unit 143 .

[0042] (4-2) Depth information acquisition unit 142 The depth information acquisition unit 142 has a function of acquiring depth information in a target space. The depth information acquisition unit 142 acquires depth information received by the communication unit 110 from the depth camera 12. The depth information acquisition unit 142 stores the acquired depth information in the storage unit 130 and outputs it to the shooting method determination unit 145.

[0043] (4-3) Shooting area determination unit 143 The shooting area determination unit 143 has a function of making a decision regarding the shooting area. For example, the shooting area determination unit 143 determines whether the shooting area is a high-definition shooting area or a low-definition shooting area. Specifically, based on the information amount information acquired by the information amount information acquisition unit 141, the shooting area determination unit 143 determines an area with a large amount of information in the target space as a high-definition shooting area, and determines an area with a small amount of information as a low-definition shooting area. The photographing area determination unit 143 outputs information indicating the result of determining the photographing area to the photographing information generation unit 146 .

[0044] Here, the determination of the shooting area will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of an omnidirectional image according to the present embodiment. Fig. 6 is a diagram showing an example of the shooting area determination according to the present embodiment.

[0045] The omnidirectional image 40a shown in FIG. 5 is an image captured of an exhibition area of ​​a museum as a target space. From the omnidirectional image 40a, it can be seen that the exhibition area is a space consisting of a ceiling, walls, and floor, and that glass cases and walls for displaying exhibits are provided. Areas that make up the space, such as the ceiling, walls, and floor, where exhibits are not displayed, are determined to be areas with little information, i.e., low-definition shooting areas, because the amount of color change is small. On the other hand, areas in the space where exhibits are displayed, such as glass cases and walls, are determined to be areas with a lot of information, i.e., high-definition shooting areas, because the amount of color change is large.

[0046] The imaging area determination result 40b shown in Fig. 6 is a visualization of the imaging area determination result in the omnidirectional image 40a shown in Fig. 5. As shown in Fig. 6, the area in the omnidirectional image 40a where exhibits such as glass cases and walls installed in the space are displayed is finely classified as a high-definition imaging area. On the other hand, the area in the omnidirectional image 40a where exhibits are not displayed such as the ceiling, walls, and floor that constitute the space are roughly classified as a low-definition imaging area. When the high-resolution and low-resolution shooting areas are not distinguished, the shooting areas in the omnidirectional image 40a are uniformly classified according to the size of the high-resolution shooting area. In this case, the omnidirectional image 40a is classified into shooting areas of 50 squares in total, 5 squares vertically and 10 squares horizontally, and 50 shot images are taken.

[0047] In contrast, when high-definition and low-definition shooting areas are distinguished as in this embodiment, the low-definition shooting areas can be grouped together so that they can be shot as large as possible, as shown in the shooting area determination result 40b. In the case of the omnidirectional image 40a in Fig. 5, the shooting area is classified into 16 squares of high-definition shooting areas and 16 squares of low-definition shooting areas, totaling 32 squares, as shown in the shooting area determination result 40b in Fig. 6, and 32 shot images are taken. In this way, by distinguishing between high-resolution and low-resolution shooting areas, the number of images to be shot can be reduced, which in turn reduces the amount of image data and processing time used to generate a panoramic image, thereby improving the efficiency of generating the panoramic image.

[0048] (4-4) Shooting setting determination unit 144 The shooting setting determination unit 144 has a function of determining the shooting setting. The shooting setting is the focal length and shooting angle of the shooting camera 13. The shooting setting determination unit 144 determines the focal length and shooting angle of the shooting camera 13 for each of the multiple shooting areas depending on whether each shooting area is a high-definition shooting area or a low-definition shooting area. When the shooting area is a high-definition shooting area, the shooting setting determination unit 144 determines the focal length and shooting angle of the shooting camera 13 so that the high-definition shooting area is shot with a telephoto lens. On the other hand, when the shooting area is a low-definition shooting area, the shooting setting determination unit 144 determines the focal length and shooting angle of the shooting camera 13 so that the low-definition shooting area is shot with a wide angle lens. The photographing setting determination unit 144 outputs information indicating the result of determining the photographing settings to the photographing information generation unit 146.

[0049] (4-5) Shooting method determination unit 145 The imaging method determination unit 145 has a function of determining an imaging method. The imaging method determination unit 145 determines whether or not to perform focus bracket imaging for each of a plurality of imaging regions based on the depth information acquired by the depth information acquisition unit 142. The imaging method determination unit 145 determines to perform focus bracket imaging for imaging regions in which a certain range of depth (depth difference) is equal to or greater than a predetermined ratio, and determines not to perform focus bracket imaging for imaging regions in which a certain range of depth is less than the predetermined ratio. The predetermined ratio may be set arbitrarily by the user, for example, 30%. The imaging method determination unit 145 outputs information indicating the imaging method determination result to the imaging information generation unit 146.

[0050] (4-6) Shooting information generating unit 146 The photographing information generating unit 146 has a function of generating photographing information. The photographing information generating unit 146 generates photographing information for each photographing area based on the result of the determination of the photographing area input from the photographing area determining unit 143, the result of the determination of the photographing setting input from the photographing setting determining unit 144, and the result of the determination of the photographing method input from the photographing method determining unit 145. The photographing information generating unit 146 stores the generated photographing information in the storage unit 130.

[0051] (4-7) Shooting control unit 147 The photographing control unit 147 has a function of controlling photographing by the photographing camera 13. The photographing control unit 147 controls photographing by the photographing camera 13 based on the photographing information stored in the storage unit 130. After photographing, the photographing control unit 147 transmits photographed image information including the photographed image and depth information to the image editing device 20 via the communication unit 110. For example, when the photographing camera 13 installed in the target space photographs a plurality of photographing areas, the photographing control unit 147 controls the focal length and photographing angle of the photographing camera 13 for each photographing area to be photographed, based on the photographing information stored in the storage unit 130. At this time, the photographing control unit 147 sets the focal length of the photographing camera 13 to the focal length indicated by the photographing information by controlling the focal length of the lens, and sets the photographing angle of the photographing camera 13 to the photographing angle indicated by the photographing information by controlling the pan or tilt of the camera platform 14.

[0052] Furthermore, when the focal center position of the lens deviates from the rotation center position of the photographing camera 13 due to the setting of the photographing angle, the photographing control unit 147 moves the photographing camera 13 horizontally to align the focal center position of the lens with the rotation center position of the photographing camera 13. Aligning the focal center position with the rotation center position (i.e., the no-parallax point) is also referred to as "no-parallax point setting" below.

[0053] Furthermore, the shooting control unit 147 checks whether the shooting method in the shooting information indicates whether focus bracket shooting is performed or not. If the shooting method indicates that focus bracket shooting is performed, the shooting control unit 147 sets focus bracket shooting and then executes focus bracket shooting. On the other hand, if the shooting method indicates that focus bracket shooting is not performed, the shooting control unit 147 sets the focal length indicated by the shooting information and then executes normal shooting.

[0054] (5) Output unit 150 The output unit 150 has a function of outputting various information. The output unit 150 is, for example, an output device provided as hardware in the imaging control device 10, such as a display device or a touch screen (touch panel), or an audio output device such as a speaker.

[0055] <3. Functional configuration of the image editing device> The configuration of the shooting control device 10 according to this embodiment has been described above. Next, the functional configuration of the image editing device 20 according to this embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the functional configuration of the image editing device 20 according to this embodiment. As shown in FIG. 7, the image editing device 20 includes a communication unit 210, an input unit 220, a storage unit 230, a control unit 240, and an output unit 250.

[0056] (1) Communications unit 210 The communication unit 210 has a function of transmitting and receiving various information. In communication with the shooting control device 10, the communication unit 210 receives a captured image and depth information. In communication with the display control device 30, the communication unit 210 transmits an edited captured image and display setting information. The display setting information is information indicating other edited captured images adjacent to the edited captured image.

[0057] (2) Input unit 220 The input unit 220 has a function of accepting inputs. The input unit 220 is, for example, an input device that the image editing device 20 has as hardware, such as a button, a touch panel, a microphone, a mouse, a keyboard, and the like.

[0058] (3) Storage section 230 The storage unit 230 has a function of storing various types of information. The storage unit 230 is configured by a storage medium provided in the image editing device 20 as hardware, such as a HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. The storage unit 230 stores, for example, a captured image before editing, depth information, a captured image after editing, display setting information, and the like.

[0059] (4) Control unit 240 The control unit 240 has a function of controlling the overall operation of the image editing device 20. The control unit 240 is realized, for example, by causing a CPU or a GPU provided as hardware in the image editing device 20 to execute a program. As shown in FIG. 7, the control unit 240 includes a captured image information acquisition unit 241, an image editing unit 242, and a display setting information generation unit 243.

[0060] (4-1) Photographed image information acquisition unit 241 The captured image information acquisition unit 241 has a function of acquiring information related to the captured image. The information related to the captured image is, for example, the captured image and depth information. The captured image information acquisition unit 241 acquires the shooting information and depth information included in the captured image information received by the communication unit 210 from the shooting control device 10. The captured image information acquisition unit 241 stores the acquired captured image and depth information in the storage unit 230 and outputs them to the image editing unit 242.

[0061] (4-2) Image Editing Department 242 The image editing unit 242 has a function of editing the captured image. The image editing unit 242 edits the captured image acquired by the captured image information acquisition unit 241. The image editing unit 242 performs editing on the captured image depending on whether the captured image is an image captured using focus bracketing (i.e., a focus bracketed image). If the captured image is not a focus bracketed image, the image editing unit 242 performs editing to embed depth information acquired by the captured image information acquisition unit 241 into the captured image. On the other hand, if the captured image is a focus bracketed image, the image editing unit 242 synthesizes the focus bracketed images and performs editing to embed depth information into the synthesized image. The image editing unit 242 outputs the edited captured image to the display setting information generating unit 243 .

[0062] (4-3) Display setting information generation unit 243 The display setting information generating unit 243 has a function of generating display setting information. The display setting information generating unit 243 identifies other edited captured images adjacent to each edited captured image edited by the image editing unit 242, and generates display setting information. The display setting information includes, for example, the ID of the captured image, and the position and size when the captured image is stitched into a panoramic image. The display setting information generating unit 243 transmits the edited captured image and display image information including the generated display setting information to the display control device 30 via the communication unit 210.

[0063] (5) Output section 250 The output unit 250 has a function of outputting various information. The output unit 250 is, for example, an output device provided as hardware in the image editing device 20, such as a display device or a touch screen (touch panel), or an audio output device such as a speaker.

[0064] <4. Functional configuration of the display control device> The configuration of the image editing device 20 according to this embodiment has been described above. Next, the functional configuration of the display control device 30 according to this embodiment will be described with reference to Fig. 8 to Fig. 12. Fig. 8 is a block diagram showing an example of the functional configuration of the display control device 30 according to this embodiment. As shown in FIG. 8, the display control device 30 includes a communication unit 310, an input unit 320, a storage unit 330, a control unit 340, and an output unit 350.

[0065] (1) Communications unit 310 The communication unit 310 has a function of transmitting and receiving various information. In communication with the image editing device 20, the communication unit 310 receives the edited captured image and display setting information. In communication with the HMD 31, the communication unit 310 receives information detected by the HMD 31 and transmits information necessary for display control of the panoramic image in the VR space.

[0066] (2) Input unit 320 The input unit 320 has a function of receiving input. The input unit 320 is configured by, for example, an input device that the display control device 30 has as hardware, such as a button, a touch panel, a microphone, a mouse, a keyboard, and the like.

[0067] (3) Storage section 330 The storage unit 330 has a function of storing various information. The storage unit 330 is configured by a storage medium provided in the display control device 30 as hardware, such as a HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. The storage unit 330 stores, for example, edited captured images, display setting information, three-dimensional objects, and the like.

[0068] (4) Control unit 340 The control unit 340 has a function of controlling the overall operation of the display control device 30. The control unit 340 is realized, for example, by causing a CPU or a GPU that the display control device 30 has as hardware to execute a program. As shown in FIG. 8, the control unit 340 includes a display image information acquisition unit 341, a three-dimensional object generation unit 342, a head detection unit 343, an attention location detection unit 344, a speed parameter setting unit 345, and a display control unit 346.

[0069] (4-1) Display image information acquisition unit 341 The display image information acquisition unit 341 has a function of acquiring information related to the display image. The information related to the display image is, for example, the edited captured image and display setting information. The display image information acquisition unit 341 acquires the edited captured image and display setting information included in the display image information received by the communication unit 310 from the image editing device 20. The display image information acquisition unit 341 stores the acquired edited captured image and display setting information in the storage unit 330, and outputs them to the three-dimensional object generation unit 342.

[0070] (4-2) Three-dimensional object generation unit 342 The three-dimensional object generating unit 342 has a function of generating a three-dimensional object. The three-dimensional object generating unit 342 generates a three-dimensional object representing a target space based on the edited captured image and the display setting information acquired by the display image information acquiring unit 341.

[0071] In generating a three-dimensional object representing a target space, the three-dimensional object generating unit 342 reads display setting information and stitches the captured images to generate a panoramic image (stitching process). The three-dimensional object generating unit 342 generates a spherical three-dimensional object with a predetermined size. The predetermined size is set arbitrarily by the user. The three-dimensional object generating unit 342 generates a three-dimensional object representing a target space by attaching the generated panoramic image (texture) to a surface inside the three-dimensional object.

[0072] (4-3) Head detection unit 343 The head detection unit 343 has a function of detecting the movement of the user's head. For example, the head detection unit 343 detects the movement of the head of the user viewing the panoramic image displayed on the HMD 31 based on the self-position estimation result by the HMD 31. The head detection unit 343 calculates the amount of head movement of the user from the difference between the self-position estimation results acquired for each frame.

[0073] (4-4) Interesting part detection unit 344 The attention location detection unit 344 has a function of detecting a user's attention location. For example, the attention location detection unit 344 detects an attention location in the panoramic image displayed on the HMD 31 that is being watched by the user based on a result of the user's line of sight detection by the HMD 31.

[0074] (4-5) Speed ​​parameter setting unit 345 The speed parameter setting unit 345 has a function of setting a speed parameter. The speed parameter is a parameter for adjusting the speed at which the image display is enlarged or reduced. The speed parameter setting unit 345 sets the speed parameter according to the distance to the subject at the detected point of interest.

[0075] The speed parameter setting unit 345 first calculates the distance to the subject based on the depth information of the captured image. The farther the calculated distance is, the slower the speed of enlargement or reduction is set by the speed parameter setting unit 345. On the other hand, the closer the calculated distance is, the faster the speed of enlargement or reduction is set by the speed parameter setting unit 345. Furthermore, in the process of enlarging or reducing the display of the panoramic image, the speed parameter setting unit 345 sets the speed parameter so that the speed of enlargement or reduction is gradually accelerated after the process starts and gradually decelerated toward the end of the process.

[0076] Here, the speed parameters will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram showing an example of the speed parameters when the point of interest according to this embodiment is at a long distance. Fig. 10 is a diagram showing an example of the speed parameters when the point of interest according to this embodiment is at a short distance.

[0077] As shown in Figures 9 and 10, the speed parameter is represented by the relationship between the amount of head movement of the user and the amount of movement of the three-dimensional object. In the graphs shown in Figures 9 and 10, the horizontal axis represents the amount of head movement, the vertical axis represents the amount of movement of the three-dimensional object, and the slope of the graph represents the speed. From the graphs in Figures 9 and 10, it can be seen that the speed at which the display of the panoramic image is enlarged or reduced is slower in the graph in Figure 9, which is a graph in which the point of interest is at a long distance, and faster in the graph in Figure 10, which is a graph in which the point of interest is at a short distance.

[0078] (4-6) Display control unit 346 The display control unit 346 has a function of controlling the display of a panoramic image in the VR space. The display control unit 346 enlarges or reduces the display of the panoramic image at a speed corresponding to the set speed parameter. The display control unit 346 can enlarge or reduce the display of the image by moving the three-dimensional object at a speed corresponding to the speed parameter. For example, the display control unit 346 enlarges the display of the panoramic image by moving the three-dimensional object to bring the point of interest closer to the user, and reduces the display of the panoramic image by moving the three-dimensional object to move the point of interest away from the user. In other words, the speed parameter is also a parameter for adjusting the moving speed of the three-dimensional object.

[0079] In this way, the display control unit 346 can change the speed at which the display of the attention point (image) is enlarged or reduced based on the speed parameter set according to the distance to the subject at the attention point. This can reduce the difference between the physical sensation and the visual amount of movement experienced by the user, making it difficult for VR sickness to occur. Furthermore, the display control unit 346 can change the speed at which the display of the point of interest is enlarged or reduced in accordance with the movement of the user's head, thereby contributing to the perception of a sense of distance even though it is a two-dimensional panorama.

[0080] Here, the movement of the three-dimensional object will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a diagram showing an example of the movement of the three-dimensional object when the display of the panoramic image according to the present embodiment is enlarged. Fig. 12 is a diagram showing an example of the movement of the three-dimensional object when the display of the panoramic image according to the present embodiment is reduced.

[0081] 11 and 12 are views of a user US wearing an HMD 31 on his head HD as viewed from above (positive direction of the Z axis). In a three-dimensional object OB shown in Fig. 11 and Fig. 12, a point P3 is the position of the user US in the VR space, and a point P4 is the focus of the user US in the panoramic image.

[0082] When the user US wishes to enlarge or reduce the display of the panoramic image displayed on the HMD 31, the user can operate it by moving the head HD. When the user US wishes to enlarge the display of the panoramic image, the user US moves the head HD, for example, in the direction of the part (point of interest) where the user wishes to enlarge the display. On the other hand, when the user US wishes to reduce the display of the panoramic image, the user US moves the head HD, for example, in the direction opposite to the direction of the part (point of interest) where the user wishes to reduce the display.

[0083] In the example shown in Fig. 11, the user US moves his / her head HD in the direction of the arrow B4-1, i.e., in the direction approaching the point of interest (point P4), in order to enlarge the display of the point of interest. In response to this movement, the display control unit 346 moves the three-dimensional object OB in the direction of the arrow B4-2, i.e., in the direction approaching the position (point P3) of the user US, in order to enlarge the display of the point of interest (point P4). As a result, the point of interest (point P4) approaches the position (point P3) of the user US, and the display of the point of interest appears to the user US to be enlarged.

[0084] 12, the user US moves his / her head HD in the direction of the arrow B5-1, i.e., in the direction away from the attention point (point P4), in order to reduce the display of the attention point. In response to this movement, the display control unit 346 moves the three-dimensional object OB in the direction of the arrow B5-2, i.e., in the direction away from the position (point P3) of the user US, in order to reduce the display of the attention point (point P4). As a result, the attention point (point P4) moves away from the position (point P3) of the user US, and the display of the attention point appears to the user US to be reduced.

[0085] (5) Output section 350 The output unit 350 has a function of outputting various information. The output unit 350 is, for example, an output device provided in the display control device 30 as hardware, such as a display device or a touch screen (touch panel), or an audio output device such as a speaker.

[0086] <5. Processing flow> The functional configuration of the display control device 30 according to this embodiment has been described above. Next, the flow of processing according to this embodiment will be described with reference to Figs.

[0087] (1) Processing flow for pre-shooting The flow of processing in pre-capturing will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the flow of processing in pre-capturing according to the present embodiment. Pre-capturing is capturing an image to be used for generating a panoramic image, and is used to prepare information necessary for capturing the image in advance.

[0088] As shown in FIG. 13, first, the information amount information acquisition unit 141 of the shooting control device 10 acquires information amount information of the target space based on the omnidirectional image received by the communication unit 110 from the omnidirectional camera 11 (step S101). Next, the depth information acquisition unit 142 of the shooting control device 10 acquires the depth information received by the communication unit 110 from the depth camera 12 (step S102). Next, the shooting area determination unit 143 of the shooting control device 10 determines the shooting areas (high-definition shooting area and low-definition shooting area) in the target space based on the information amount information acquired by the information amount information acquisition unit 141 (step S103).

[0089] Next, the shooting setting determination unit 144 of the shooting control device 10 checks whether the shooting area is a high-definition shooting area (or a low-definition shooting area) (step S104). If the shooting area is a high-definition shooting area (step S104 / YES), the process proceeds to step S105. On the other hand, if the shooting area is not a high-definition shooting area (a low-definition shooting area) (step S104 / NO), the process proceeds to step S107.

[0090] When the process proceeds to step S105, the imaging method determination unit 145 checks whether or not there is a predetermined depth difference in the imaging area based on the depth information (step S105). When the depth difference is equal to or greater than the predetermined ratio (step S105 / YES), the process proceeds to step S106. On the other hand, when the depth difference is less than the predetermined ratio (step S105 / NO), the process proceeds to step S107.

[0091] When the process proceeds to step S106, the photographing method determination unit 145 determines to perform focus bracket photographing on the photographing area (step S106). In this case, the photographing angle and focal length of the photographing area to be subjected to focus bracket photographing are set according to the focus bracket photographing. After the photographing method is determined, the process proceeds to step S109.

[0092] When the process proceeds to step S107, the shooting setting determination unit 144 determines the shooting angles of the low-definition shooting area and the high-definition shooting area where focus bracket shooting is not performed (step S107). In this case, the shooting angle that matches the low-definition shooting area or the high-definition shooting area is set from among the shooting angles prepared in advance. Next, the shooting setting determination unit 144 determines the focal length of the low-definition shooting area and the high-definition shooting area where focus bracket shooting is not performed (step S108). In this case, a focal length that matches the low-definition shooting area or the high-definition shooting area is set from among the focal lengths prepared in advance. After the focal length is set, the process proceeds to step S109.

[0093] When the process proceeds to step S109, the photographing information generating unit 146 generates photographing information indicating the results of the determinations made by the photographing area determining unit 143, the photographing setting determining unit 144, and the photographing method determining unit 145 (step S109). The photographing information generating unit 146 stores the generated photographing information in the storage unit 130. If the shooting information of all the shooting areas has not been generated (step S110 / NO), the process is repeated from step S104, and the shooting control device 10 generates shooting information for the remaining shooting areas. If the shooting information of all the shooting areas has been generated (step S110 / YES), the process ends.

[0094] (2) Processing flow during actual shooting The flow of processing in actual shooting will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the flow of processing in actual shooting according to this embodiment. The actual shooting is shooting for preparing shot images to be used for generating a panoramic image, using information prepared in advance shooting.

[0095] As shown in FIG. 14, first, the imaging control unit 147 of the imaging control device 10 acquires imaging information of an imaging area to be imaged from the storage unit 130 (step S201). Next, the photographing control unit 147 controls the camera platform 14 to set the photographing angle of the photographing camera 13 to the photographing angle indicated by the photographing information (step S202). Next, the photographing control unit 147 controls the camera platform 14 to align the focal center position and the rotation center position of the photographing camera 13, and sets the no-parallax point (step S203). Note that the no-parallax point needs to be set only when the focal center position and the rotation center position are misaligned due to the setting of the photographing angle.

[0096] Next, the shooting control unit 147 checks whether the shooting area is a target of focus bracket shooting by referring to the shooting method of the shooting information (step S204). If it is a target of focus bracket shooting (step S204 / YES), the process proceeds to step S205. On the other hand, if it is not a target of focus bracket shooting (step S204 / NO), the process proceeds to step S206.

[0097] When the process proceeds to step S205, the shooting control unit 147 sets the shooting camera 13 for focus bracket shooting (step S205). After the setting, the process proceeds to step S207. When the process proceeds to step S206, the photographing control unit 147 sets the focal length indicated by the photographing information for the photographing camera 13 (step S206). After the setting, the process proceeds to step S207.

[0098] When the process proceeds to step S207, the photographing control unit 147 causes the photographing camera 13 to photograph the photographing area (step S207). If the imaging of all the imaging regions has not been completed (step S208 / NO), the process is repeated from step S201 to image the remaining imaging regions. If the imaging of all the imaging regions has been completed (step S208 / YES), the process proceeds to step S209. When the process proceeds to step S209, the shooting control unit 147 transmits the shot image information including the shot image and the depth information to the image editing device 20 via the communication unit 110 (step S209), and ends the process.

[0099] (3) Image editing process The flow of processing in image editing will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the flow of processing in image editing according to this embodiment.

[0100] As shown in FIG. 15, first, the captured image information acquisition unit 241 of the image editing device 20 acquires the captured image information received by the communication unit 210 from the shooting control device 10 (step S301). Next, the image editing unit 242 of the image editing device 20 checks whether or not there is a shooting area that has been subjected to focus bracket shooting, based on the shooting image information acquired by the shooting image information acquisition unit 241 (step S302). If there is a shooting area that has been subjected to focus bracket shooting (step S302 / YES), the process proceeds to step S303. On the other hand, if there is no shooting area that has been subjected to focus bracket shooting (step S302 / NO), the process proceeds to step S304.

[0101] When the process proceeds to step S303, the image editing unit 242 synthesizes the focus-bracketed images (step S303). After the synthesis, the process proceeds to step S304. When the process proceeds to step S304, the image editing unit 242 performs editing to embed the depth information acquired by the captured image information acquisition unit 241 into the captured image (including the composite image of the focus bracketed images) (step S304). After the editing, the display setting information generating unit 243 of the image editing device 20 generates display setting information for each captured image edited by the image editing unit 242 (step S305). The display setting information generating unit 243 stores the generated display setting information in the storage unit 230. After the generation, the display setting information generation unit 243 transmits the display image information including the edited photographed image and the generated display setting information to the display control device 30 via the communication unit 210 (step S306).

[0102] (4) Processing flow for viewing images The flow of processing in image viewing will be described with reference to Fig. 16. Fig. 16 is a flowchart showing an example of the flow of processing in image viewing according to this embodiment.

[0103] As shown in FIG. 16, first, the display control device 30 starts up the viewer (step S401). Next, the three-dimensional object generating unit 342 of the display control device 30 reads the display setting information included in the display image information acquired by the display image information acquiring unit 341 of the display control device 30 (step S402). Next, the three-dimensional object generator 342 performs a stitching process on the captured images included in the display image information (step S403). Through the stitching process, a panoramic image of the target space is generated. Next, the three-dimensional object generator 342 generates a three-dimensional object (step S404). Next, the three-dimensional object generating unit 342 pastes the generated panoramic image (texture) onto the inner surface of the generated three-dimensional object (step S405). Next, the display control unit 346 of the display control device 30 displays the generated panoramic image of the inside of the three-dimensional object on the HMD 31 (step S406). The user wears the HMD 31 on which the panoramic image is displayed (step S407).

[0104] After putting it on, the user starts viewing the panoramic image (step S408). After the user starts viewing, the head detection unit 343 of the display control device 30 detects the movement of the user's head (step S409). At this time, the head detection unit 343 calculates the amount of head movement of the user based on the detection result acquired for each frame. The head detection unit 343 checks whether the calculated amount of head movement of the user is equal to or greater than a predetermined amount (step S410). If it is equal to or greater than the predetermined amount (step S410 / YES), the process proceeds to step S411. On the other hand, if it is less than the predetermined amount (step S410 / NO), the process repeats from step S409.

[0105] When the process proceeds to step S411, the attention location detection unit 344 of the display control device 30 detects the user's attention location based on the user's gaze detection result by the HMD 31 (step S411). Next, the speed parameter setting unit 345 of the display control device 30 sets a speed parameter according to the distance to the subject at the detected point of interest (step S412). Next, the display control unit 346 enlarges or reduces the display of the panoramic image by moving the three-dimensional object at a speed according to the set speed parameter (step S413). After the three-dimensional object is moved, the process repeats from step S409.

[0106] As described above, the image display system 3 according to this embodiment includes an attention point detection unit 344 that detects an attention point in an image that is being focused on by a user viewing an image displayed on an HMD 31 (display device), a speed parameter setting unit 345 that sets a speed parameter according to the distance to a subject at the detected attention point, and a display control unit 346 that enlarges or reduces the display of the image at a speed according to the set speed parameter.

[0107] With this configuration, the image display system 3 can change the speed at which the display of the attention point (image) is enlarged or reduced depending on the distance to the subject at the attention point. This can reduce the difference between the physical sensation and the visual movement amount experienced by the user, making it less likely for VR sickness to occur. Therefore, the image display system 3 according to the present embodiment can further suppress VR sickness.

[0108] <6. Variations> The embodiments of the present invention have been described above. Next, modified examples of the embodiments of the present invention will be described. Each modified example described below may be applied alone to the embodiments of the present invention, or may be applied in combination to the embodiments of the present invention. Furthermore, each modified example may be applied in place of the configuration described in the embodiments of the present invention, or may be applied in addition to the configuration described in each embodiment of the present invention.

[0109] In the above embodiment, an example has been described in which the omnidirectional camera 11, the depth camera 12, and the photographing camera 13 are mounted on the camera platform 14 and photographing or measuring is performed based on the control of the photographing control device 10, but the present invention is not limited to such an example. For example, the omnidirectional camera 11, the depth camera 12, and the photographing camera 13 may be mounted on the camera platform 14 and photographing or measuring may be performed based on manual operation by a user.

[0110] In the above embodiment, an example in which the depth information is acquired using the depth camera 12 has been described, but the present invention is not limited to such an example. For example, the depth information may be acquired based on a plurality of captured images obtained by capturing images of one capturing area using the capturing camera 13 while shifting the focus.

[0111] In the above-described embodiment, an example has been described in which the display control of the panoramic image (three-dimensional object OB) in the HMD 31 is performed by the display control device 30, but the present invention is not limited to such an example. For example, part or all of the display control of the panoramic image may be performed by the HMD 31. In this case, the HMD 31 is provided with part or all of the functions of the display control device 30 in the above-described embodiment, thereby enabling the HMD 31 to execute part or all of the display control.

[0112] In the above embodiment, the shooting control device 10 determines the high-definition shooting area and the low-definition shooting area in the target space based on the amount of information in the target space, but the present invention is not limited to this example. For example, the shooting control device 10 may determine the high-definition shooting area and the low-definition shooting area in the target space using an object recognition algorithm such as machine learning. The object recognition algorithm recognizes and segments objects (e.g., people, buildings, paintings, etc.) present in the target space from the omnidirectional images captured of the target space, and determines whether the shooting area is a high-definition shooting area or a low-definition shooting area based on the results. For example, the object recognition algorithm determines a shooting area including an object classified as being shot in high definition to be a high-definition shooting area, and determines a shooting area including an object classified as being shot in low definition to be a low-definition shooting area. By using an object recognition algorithm, the shooting control device 10 may be able to determine the shooting area with higher accuracy than when the shooting area is determined based on the amount of information. For example, assume that a light blue airplane is flying in the sky of the target space, and it is desired to shoot the shooting area including the airplane as a high-definition shooting area. In this case, if the shooting control device 10 uses a method of determination based on the amount of information, the difference between the color of the sky and the color of the airplane is small, so the difference in color information between pixels is not large, and the shooting area including the light blue airplane is determined to be an area with low information amount, and is determined to be a low-definition shooting area. On the other hand, if the shooting control device 10 uses a method of determination based on an object recognition algorithm, the light blue airplane can be recognized as an object by machine learning the image of the airplane, and the shooting area including the light blue airplane can be determined to be a high-definition shooting area. When an object recognition algorithm is used, the photography control device 10 determines the target area using, for example, a machine-learned model (trained model) for the object recognition algorithm. In this case, the photography area determination unit 143 of the photography control device 10 determines whether the photography area is a high-definition photography area or a low-definition photography area using the trained model stored in the storage unit 130. In addition, the method using the object recognition algorithm may be used alone or in combination with the method using the information amount. When used alone, the shooting control device 10 does not need to include the information amount information acquisition unit 141.

[0113] The above describes the modified embodiment of the present invention. In addition, some or all of the functions of the panoramic VR system 1000, the shooting system 1, the image editing system 2, the image display system 3, the shooting control device 10, the image editing device 20, and the display control device 30 in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. In addition, the "computer system" here includes hardware such as an OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in a computer system, or may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0114] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0115] 1...photography system, 2...image editing system, 3...image display system, 10...photography control device, 11...omnidirectional camera, 12...depth camera, 13...photography camera, 14...platform, 20...image editing device, 30...display control device, 110...communication unit, 120...input unit, 130...storage unit, 140...control unit, 141...information amount information acquisition unit, 142...depth information acquisition unit, 143...photography area determination unit, 144...photography setting determination unit, 145...photography method determination unit, 146...photography information generation unit, 147...photography control unit, 150...output unit, 2 10...communication unit, 220...input unit, 230...storage unit, 240...control unit, 241...captured image information acquisition unit, 242...image editing unit, 243...display setting information generation unit, 250...output unit, 310...communication unit, 320...input unit, 330...storage unit, 340...control unit, 341...display image information acquisition unit, 342...three-dimensional object generation unit, 343...head detection unit, 344...interest area detection unit, 345...speed parameter setting unit, 346...display control unit, 350...output unit, 1000...panoramic VR system, NW...network

Claims

1. an attention location detection unit that detects an attention location in an image displayed on a display device that is being watched by a user; a speed parameter setting unit that sets a speed parameter according to a distance to a subject at the detected point of interest; a display control unit that enlarges or reduces a display of the image at a speed corresponding to the set speed parameter; An image display system comprising:

2. the speed parameter setting unit calculates the distance based on depth information included in the image, and sets the speed parameter so that the speed becomes slower as the calculated distance becomes longer, and sets the speed parameter so that the speed becomes faster as the calculated distance becomes shorter.

2. The image display system according to claim 1.

3. the speed parameter setting unit sets the speed parameter such that, in a process of enlarging or reducing the display of the image, the speed is gradually increased after the process starts and the speed is gradually decreased toward the end of the process.

3. The image display system according to claim 1 or 2.

4. the image is attached to an interior surface of a spherical three-dimensional object; the display device displays the image so that the user is viewing the image inside the three-dimensional object; the display control unit enlarges or reduces the display of the image by moving the three-dimensional object at a speed corresponding to the speed parameter.

2. The image display system according to claim 1.

5. the display control unit moves the three-dimensional object to bring the attention point closer to the user to enlarge the display of the image, and moves the three-dimensional object to move the attention point away from the user to reduce the display of the image.

5. The image display system according to claim 4.

6. an attention location detection step of detecting an attention location in an image displayed on a display device that is being watched by a user; a speed parameter setting step of setting a speed parameter according to a distance to a subject at the detected point of interest; a display control step of enlarging or reducing the display of the image at a speed according to the set speed parameter; 23. A computer-implemented image display method comprising:

7. Computer, an attention location detection means for detecting an attention location in an image displayed on a display device that is being watched by a user; a speed parameter setting means for setting a speed parameter according to a distance to a subject at the detected point of interest; a display control means for enlarging or reducing the display of the image at a speed corresponding to the set speed parameter; A program to function as a

Images