Image processing apparatus, image processing method, and program product

Abstract

A display device (10) as an example of the image processing device includes an output unit that outputs a part of an image including recommended viewpoint information as a display image to a display unit, and a transformation control unit that transforms a display area of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information. The output unit outputs a part of the image to the display unit based on the transformed display area.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program. Background Art

[0002] The omnidirectional media application format (OMAF) standard of MPEG-DASH (Dynamic Adaptive Streaming over HTTP) defines a method for transmitting recommended viewpoint information together with data of a wide-viewpoint image having a viewing angle of 360° or less. In addition, for example, Patent Document 1 discloses the display of a recommended viewpoint.

[0003] Citation List

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-194784 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] When a recommended viewpoint is displayed, only the recommended viewpoint is viewed in a wide-viewpoint image. On the other hand, if a user can freely view an image, the user cannot see the position of the recommended viewpoint. In this case, it is conceivable to display the recommended viewpoint with a border or the like, but the world depicted in the image may shake at that part.

[0008] Therefore, the present disclosure aims to provide an image processing apparatus, an image processing method, and a program that enable the combined use of a recommended viewpoint and a free viewpoint without displaying the position of the recommended viewpoint with a border or the like.

[0009] Solutions to the Problems

[0010] To solve the above problems, an image processing apparatus according to an embodiment of the present disclosure includes: an output unit that outputs a part of an image including recommended viewpoint information as a display image to a display unit; and a transformation control unit that changes a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information. In this image processing apparatus, the output unit outputs a part of the image to the display unit based on the changed display range. Brief Description of the Drawings

[0011] Figure 1 is a diagram showing an example configuration of an image distribution system according to a first embodiment.

[0012] Figure 2A is a diagram showing an example spherical surface corresponding to an image captured by a sphere.

[0013] Figure 2BIt is a schematic diagram showing an example coordinate system of a projected image.

[0014] Figure 2C It is a diagram showing an example of recommended viewpoint information.

[0015] Figure 3 It is a block diagram showing an example functional configuration of a display device according to the first embodiment.

[0016] Figure 4A It is a diagram showing an example projected image.

[0017] Figure 4B It is a diagram showing an example display range.

[0018] Figure 4C It is a diagram showing an example display image.

[0019] Figure 5 It is a graph showing an example method for calculating a return movement amount.

[0020] Figure 6 It is a graph showing an example method for calculating a return movement amount.

[0021] Figure 7 It is a flowchart showing the process of display processing according to the first embodiment.

[0022] Figure 8 It is a graph showing an example method for setting a threshold.

[0023] Figure 9 It is a flowchart showing the process of display processing of an example application according to the first embodiment.

[0024] Figure 10 It is a block diagram showing an example functional configuration of a display device according to the second embodiment.

[0025] Figure 11 It is a graph showing an example method for correcting an operation movement amount.

[0026] Figure 12 It is a graph showing an example method for correcting an operation movement amount.

[0027] Figure 13 It is a flowchart showing the process of display processing according to the second embodiment.

[0028] Figure 14 It is a block diagram showing an example functional configuration of a display device according to the third embodiment.

[0029] Figure 15 It is a flowchart showing the process of display processing according to the third embodiment.

[0030] Figure 16 It is a block diagram showing an example functional configuration of a display device according to a fourth embodiment.

[0031] Figure 17 It is a diagram showing an example of the distance used in fraction calculation.

[0032] Figure 18 It is a diagram showing examples of the distance and weight used in fraction calculation.

[0033] Figure 19 It is a flowchart showing the process of display processing according to the fourth embodiment.

[0034] Figure 20 It is a block diagram showing an example configuration of computer hardware. Detailed Description of the Invention

[0035] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in each of the embodiments described below, the same components are denoted by the same reference numerals, and their repeated description will be omitted.

[0036] In addition, the present disclosure will be explained in the following order.

[0037] 1. First Embodiment

[0038] 1-1. Image Distribution System

[0039] 1-2. Distribution-Side Device

[0040] 1-3. Receiving-Side Device

[0041] 1-3-1. Receiving Unit

[0042] 1-3-2. Gyro Sensor

[0043] 1-3-3. Transition Control Unit

[0044] 1-3-4. Field of View Rendering Unit

[0045] 1-3-5. Display Unit

[0046] 1-4. Details of the Transition Control Unit

[0047] 1-4-1. Operation Movement Amount Calculation Unit

[0048] 1-4-2. Return Movement Amount Calculation Unit

[0049] 1-4-3. Combined Movement Amount Calculation Unit

[0050] 1-4-4. Viewpoint Position Calculation Unit

[0051] 1-5. Processing Procedure to be Executed by the Display Device

[0052] One aspect of the effect

[0053] 1-7. Example application of the first embodiment

[0054] 1-7-1. Process in the processing according to the example application

[0055] 2. Second embodiment

[0056] 2-1. Example functional configuration of the display device

[0057] 2-2. Transformation control unit

[0058] 2-2-1. Calibration unit

[0059] 2-3. Processing procedure to be executed by the display device

[0060] 2-4. One aspect of the effect

[0061] 3. Third embodiment

[0062] 3-1. Example functional configuration of the display device

[0063] 3-2. Transformation control unit

[0064] 3-3. Processing procedure to be executed by the display device

[0065] 3-4. One aspect of the effect

[0066] 4. Fourth embodiment

[0067] 4-1. Example functional configuration of the display device

[0068] 4-2. Transformation control unit

[0069] 4-2-1. Fraction calculation unit

[0070] 4-2-2. Recommended viewing point selection unit

[0071] 4-3. Processing procedure to be executed by the display device

[0072] 4-4. One aspect of the effect

[0073] 5. Variation

[0074] 5-1. Feedback other than visual

[0075] 5-2. Execution conditions for guiding induction control

[0076] 5-3. Device for implementing functions

[0077] 5-4. Other variations

[0078] 6. Hardware Configuration

[0079] <<1. First Embodiment>>

[0080] <1-1. Image Distribution System>

[0081] Figure 1 is a diagram showing an example configuration of an image distribution system according to the first embodiment. As Figure 1 shown, the image distribution system 1 includes a distribution-side device and a display device 10 as a receiving-side device. Examples of the installation of the display device 10 include a head-mounted display (HMD), a tablet computer, etc.

[0082] <1-2. Distribution-Side Device>

[0083] The distribution-side device includes a multi-camera 3, a wide-view image conversion unit 5, an encoder 7, and a distribution server 9.

[0084] The multi-camera 3 acquires image data of a spherical captured image. For example, the multi-camera 3 images in a back-to-back manner with two cameras and acquires a front image and a back image each having a wide view angle of 180° or more captured as a spherical captured image using a fish-eye lens.

[0085] The wide-view image conversion unit 5 performs planar packaging on the spherical captured image acquired by the multi-camera 3 and acquires a rectangular projection image (projection picture). This projection image corresponds to an image with a wide view angle having a view angle of 360° or less. In this case, for example, equirectangular projection or the like is selected as the format type of the projection image. Note that the wide-view image conversion unit 5 scales the projection image as needed to obtain a projection image with a predetermined resolution.

[0086] The encoder 7, for example, performs encoding such as HEVC on the image data of the projection image provided from the wide-view image conversion unit 5 to obtain encoded image data, and generates a video stream including the encoded image data. Here, the encoder 7 inserts recommended viewpoint information frame by frame into the video stream.

[0087] For example, the recommended viewpoint (RVP) is automatically set within the range including the performer estimated by a position sensor or through image recognition, or is set, for example, within the range manually specified by a director. Here, not necessarily only one recommended viewpoint is set, but multiple recommended viewpoints can be set.

[0088] Figure 2A Shows the spherical surface corresponding to the spherical captured image. Figure 2B Schematically shows a rectangular projection image in the case where the format type is equirectangular projection. The center of this projection image is (0, 0).

[0089] The recommended viewpoint information includes, for example, Figure 2C the frame number, central angle information, azimuth angle information, elevation angle information, and recommended viewpoint number as shown in Figure 2A .

[0090] Return reference Figure 1 , and the distribution server 9 transmits the video stream generated by the encoder 7 to the display device 10 as distribution data. Note that such distribution can be either broadcast or communication.

[0091] <1-3. Receiving-side device>

[0092] Figure 3 is a block diagram showing an example functional configuration of a display device according to the first embodiment. Figure 3 Schematically shows blocks corresponding to the functions of the display device 10. As Figure 3 shown, the display device 10 includes a receiving unit 11, a decoder 12, a field-of-view rendering unit 13, a gyro sensor 14, a transition control unit 15, and a display unit 16. Note that the field-of-view rendering unit 13 corresponds to an example of the output unit.

[0093] <1-3-1. Receiving unit>

[0094] The receiving unit 11 receives the video stream as distribution data transmitted from the transmission-side device. The decoder 12 decodes the video stream received by the receiving unit 11 to obtain a projection image (image data). The decoder 12 also obtains the recommended viewpoint information inserted for each frame in the received video stream.

[0095] <1-3-2. Gyro sensor>

[0096] The gyro sensor 14 detects changes in the rotation angle of the device including the gyro sensor 14 (in this case, the device is the display device 10). Such a detection output can correspond to sight line operation information, such as a change in the rotation angle of the display device 10 causing the user wearing the HMD to rotate the neck or causing the user to rotate the tablet. Note that the present invention can also be applied to a usage scenario where the user performs an operation such as a slide on the touch panel to generate a viewpoint operation. In this case, the information about the operation on the touch panel can correspond to the viewpoint operation information.

[0097] <1-3-3. Transition control unit>

[0098] The transformation control unit 15 changes the display range of the display image to be displayed on the display unit 16 in the projection image based on the recommended viewpoint information of the next frame from the decoder 12 and the viewpoint operation information from the gyro sensor 14. In the example, this change in the display range can be achieved by calculating the viewpoint position of the next frame for each frame. For example, with the viewpoint position of the next frame as the optical center, the transformation control unit 15 can determine the display range of the next frame as the field of view defined depending on the horizontal viewing angle and the vertical viewing angle or depending on the diagonal viewing angle (each of which is set to the size of the display image).

[0099] <1-3-4. Field of view rendering unit>

[0100] The field of view rendering unit 13 intercepts and renders the image data of the display range of the next frame calculated by the transformation control unit 15 from the projection image data of the next frame obtained by the decoder 12, and obtains the display image data corresponding to the viewpoint position of the next frame.

[0101] <1-3-5. Display unit>

[0102] The display unit 16 displays an image based on the display image data obtained by the field of view rendering unit 13 for each frame.

[0103] Figure 4A is a diagram showing an example projection image. Figure 4B is a diagram showing an example display range. Figure 4C is a diagram showing an example display image. As Figure 4B shown, only as an example, the display range of the next frame corresponding to the viewpoint position of the next frame is set on Figure 4A the projection image shown. In the case of setting such a display range, as Figure 4C shown, the display image corresponding to the display range is intercepted from the projection image and then displayed on the display unit 16.

[0104] <1-4. Details of the transformation control unit>

[0105] As Figure 3 shown, the transformation control unit 15 includes an operation movement amount calculation unit 15A, a return movement amount calculation unit 15B, a combined movement amount calculation unit 15C, and a viewpoint position calculation unit 15D.

[0106] <1-4-1. Operation movement amount calculation unit>

[0107] The operation movement amount calculation unit 15A is a processing unit that calculates the movement amount of the viewpoint position corresponding to the displayed image according to the viewpoint operation based on the viewpoint operation information. Hereinafter, the movement amount of the viewpoint position corresponding to the displayed image according to the viewpoint operation is also referred to as the "operation movement amount" in some cases.

[0108] As an example, the operation movement amount calculation unit 15A calculates the above-mentioned operation movement amount based on the line-of-sight operation information output by the gyro sensor 14. More specifically, for each frame, the angular velocities of the three axes of pitch, roll, and yaw are input from the gyro sensor 14 to the operation movement amount calculation unit 15A. According to such input, the operation movement amount calculation unit 15A performs the following processing for each of the three axes. That is, the operation movement amount calculation unit 15A calculates the rotation angle by integrating the angular velocity output from the gyro sensor 14. By subtracting the rotation angle calculated in the previous frame from the rotation angle calculated in the current frame in this way, the operation movement amount calculation unit 15A calculates the operation movement amount of the next frame.

[0109] <1-4-2. Return movement amount calculation unit>

[0110] The return movement amount calculation unit 15B is a processing unit that calculates the movement amount for returning the viewpoint position corresponding to the displayed image to the viewpoint position corresponding to the recommended viewpoint information. Hereinafter, the movement amount for returning the viewpoint position corresponding to the displayed image to the recommended viewpoint is also referred to as the "return movement amount" in some cases.

[0111] As an example, the return movement amount calculation unit 15B calculates the above-mentioned return movement amount based on the recommended viewpoint information output by the decoder 12. More specifically, the return movement amount calculation unit 15B also performs the following processing for each of the three axes of pitch, roll, and yaw. That is, the return movement amount calculation unit 15B calculates the above-mentioned return movement amount based on the angular difference of the line-of-sight direction from the recommended viewpoint to the viewpoint position of the current frame. For example, as the angular difference of the line-of-sight direction from the recommended viewpoint to the viewpoint position of the current frame becomes larger, the return movement amount calculation unit 15B sets the speed of returning the viewpoint position to the recommended viewpoint to be higher.

[0112] Figure 5 and Figure 6 are graphs each showing an example method for calculating the return movement amount. Figure 5 and Figure 6 In the graphs shown in, the horizontal axis indicates the angle relative to the recommended viewpoint, and the horizontal axis indicates the speed of returning the viewpoint position to the recommended viewpoint. The "speed" mentioned here is only an example and refers to the movement amount per frame, and can be determined by setting the number of frames required before returning to the recommended viewpoint. As Figure 5As shown, the above return movement amount can be calculated according to a function, where as the angular difference in the line-of-sight direction from the recommended viewing point to the viewing point position in the current frame increases, the speed at which the viewing point position returns to the recommended viewing point linearly increases. In another example, the above return movement amount can also be calculated according to a function, where as the angular difference in the line-of-sight direction from the recommended viewing point to the viewing point position in the current frame increases, the speed at which the viewing point position returns to the recommended viewing point monotonically increases, and the monotonic increase ratio decreases.

[0113] Since the guidance control for returning the viewing point position to the recommended viewing point is incorporated in this way in the transition of the display range, the recommended viewing point can be visually feedback without displaying various marks such as borders and arrows indicating the position of the recommended viewing point. In addition, since the sense of distance to the recommended viewing point can be transmitted by calculating the above return movement amount according to the Figure 5 and Figure 6 functions shown, VR discomfort can be reduced and the sense of strangeness when guiding to the recommended viewing point can be reduced. In addition, since the above return movement amount is calculated according to the Figure 6 functions shown, the sense of distance to the recommended viewing point can be transmitted more smoothly compared to the Figure 5 cases shown.

[0114] Note that examples of using the same function for the three axes of pitch, roll, and yaw are described herein, but different functions can be used for the three axes. For example, for the roll axis, the above return movement amount can be set to zero regardless of the magnitude of the angular difference in the line-of-sight direction from the recommended viewing point to the viewing point position in the current frame. Alternatively, a function that multiplies different weights for the three corresponding axes can be used.

[0115] In addition, although Figure 5 and Figure 6 illustrate example functions for determining the return movement amount, the return movement amount does not necessarily have to be calculated according to a function. For example, instead of a function, a look-up table, etc. can of course be used, where the speed at which the viewing point position returns to the recommended viewing point is associated with the corresponding level of the rotation angle in the line-of-sight direction from the recommended viewing point to the viewing point position in the current frame.

[0116] <1-4-3. Combined movement amount calculation unit>

[0117] The combined movement amount calculation unit 15C is a processing unit that calculates the combined movement amount by combining the operation movement amount and the return movement amount. Only as an example, the combined movement amount calculation unit 15C calculates the sum of the operation movement amount calculated by the operation movement amount calculation unit 15A and the return movement amount calculated by the return movement amount calculation unit 15B for each of the three axes of roll and yaw. Therefore, the combined movement amount obtained by combining the operation movement amount and the return movement amount can be calculated.

[0118] <1-4-4. Viewpoint position calculation unit>

[0119] The viewpoint position calculation unit 15D is a processing unit that calculates the viewpoint position of the next frame based on the combined movement amount. By way of example only, the viewpoint position calculation unit 15D calculates the viewpoint position of the next frame by adding the angles of the line-of-sight directions corresponding to the combined movement amounts calculated by the combined movement amount calculation unit 15C for each of the three axes of roll and yaw to the line-of-sight direction of the current frame. After that, with the viewpoint position of the next frame as the optical center, the viewpoint position calculation unit 15D can determine the display range of the next frame as depending on the horizontal viewing angle and the vertical viewing angle, or depending on the diagonal viewing angle (each of which is set to the size of the display image) defined field of view. The display range of the next frame determined in this way is input to the field-of-view rendering unit 13.

[0120] <1-5. Processing procedures to be executed by the display device>

[0121] Figure 7 is a flowchart showing the process of the display process according to the first embodiment. This process starts in response to a video clip reproduction start operation performed by the user by way of example only. When the process starts in this way, the display device 10 displays the recommended viewpoint as the display range, which is the range (recommended range) corresponding to the RVP, as Figure 7 shown in (step S101).

[0122] After that, the subsequent processes in steps S102 to S104 are executed for each of the three axes of pitch, roll, and yaw.

[0123] That is, the operation movement amount calculation unit 15A calculates the above operation movement amount based on the line-of-sight operation information output by the gyro sensor 14 (step S102). In addition, the return movement amount calculation unit 15B calculates the above return movement amount based on the recommended viewpoint information output by the decoder 12 (step S103).

[0124] After that, the combined movement amount calculation unit 15C calculates the combined movement amount by adding the operation movement amount calculated in step S102 and the return movement amount calculated in step S103 for each of the three axes of roll and yaw (step S104).

[0125] Subsequently, the viewpoint position calculation unit 15D calculates the viewpoint position of the next frame by adding the angles of the line-of-sight directions corresponding to the combined movement amounts calculated for each of the three axes of roll and yaw in step S104 to the line-of-sight direction of the current frame (step S105).

[0126] Then, the field-of-view rendering unit 13 causes the display unit 16 to display a display image, and sets the field of view corresponding to the viewpoint position of the next frame calculated in step S105 as the display range of the projection image (step S106).

[0127] Thereafter, the processes in steps S102 to S106 described above are repeated until video reproduction or distribution is completed ("No" in step S107). Thereafter, when video reproduction or distribution is completed ("Yes" in step S107), the process ends.

[0128] <1-6. Aspect of the effect>

[0129] As described above, the display device 10 of the present disclosure moves the display range of the projection image based on the line-of-sight operation information and the recommended viewpoint information, and displays a display image corresponding to the moved display range on the display unit 16. Therefore, with the display device 10 of the present disclosure, the recommended viewpoint can be used in combination with the free viewpoint without necessarily indicating the position of the recommended viewpoint with a border or the like.

[0130] <1-7. Example application of the first embodiment>

[0131] From the aspect of reducing vibration near the recommended viewpoint, the display device 10 of the present disclosure can perform threshold processing for not performing a line-of-sight operation when the above-described operation movement amount is equal to or less than a predetermined threshold. Figure 8 It is a graph showing an example method for setting a threshold.

[0132] Figure 8 In the graph shown, the horizontal axis indicates the angle relative to the recommended viewpoint, and the vertical axis indicates the threshold for determining not to perform a line-of-sight operation. As Figure 8 shown, the smaller the angular difference between the line-of-sight direction from the recommended viewpoint to the viewpoint position of the current frame, the higher the threshold for determining not to perform a line-of-sight operation is set. In other words, the larger the angular difference between the line-of-sight direction from the recommended viewpoint to the viewpoint position of the current frame, the lower the threshold for determining not to perform a line-of-sight operation is set. By setting such a threshold, vibration near the recommended viewpoint can be reduced.

[0133] <1-7-1. Process in the processing according to the example application>

[0134] Figure 9 It is a flowchart showing the process in the display processing according to the example application of the first embodiment. In Figure 9 the flowchart shown, the same step numbers are assigned to the steps that perform the same content as the processing in the flowchart shown in Figure 7 but different step numbers are assigned to the steps that perform different content.

[0135] Figure 9 The display processing shown in Figure 7 The difference shown in FIG. 1 is that the processing in steps S201 to S203 is added. Specifically, as an example only, step S201 may be performed after the processing in step S103.

[0136] For example, in step S201 , the combined operation amount calculation unit 15C determines whether the operation movement amount calculated in step S102 exceeds a predetermined threshold value, such as a threshold value determined by the angular difference of the line of sight direction from the recommended viewpoint to the viewpoint position of the current frame.

[0137] If the operation movement amount exceeds the threshold here, or if the operation movement amount is neither equal to nor less than the threshold ("Yes" in step S201), the combined operation amount calculation unit 15C calculates the combined movement amount by adding the operation movement amount calculated in step S102 for each of the three axes of roll and yaw and the return movement amount calculated in step S103 (step S202).

[0138] On the other hand, if the operation movement amount does not exceed the threshold value, or if the operation movement amount is equal to or less than the threshold value ("No" in step S201), the combined operation amount calculation unit 15C determines that the line of sight operation is not performed, or the operation movement amounts of the three axes of roll and yaw are 0. That is, the combined operation amount calculation unit 15C calculates the return movement amounts of the three axes of roll and yaw as the combined movement amount (step S203).

[0139] Thereafter, based on the combined movement amount calculated in step S202 or step S203 , the processing in the subsequent step S105 is performed.

[0140] According to the exemplary application of the first embodiment, vibrations in the vicinity of the recommended viewpoint can be reduced.

[0141] <<2. Second embodiment>>

[0142] In the above example of the above first embodiment, the sense of distance from the recommended viewpoint is transmitted by adjusting the return movement amount. However, it is not necessary to perform the guidance control for returning the viewpoint position to the recommended viewpoint to transmit the sense of distance from the recommended viewpoint. In view of this, an example case of transmitting the sense of distance from the recommended viewpoint by limiting the above operation movement amount will now be described as a second embodiment.

[0143] <2-1. Example Functional Configuration of Display Device>

[0144] Figure 10 1 is a block diagram showing an example functional configuration of a display device according to the second embodiment. Figure 10 As shown in FIG. , the display device 20 and Figure 3The display device 10 shown in [the figure] is different in that it includes a transition control unit 21 that performs processing that is partially different in content from the processing performed by the transition control unit 15.

[0145] <2-2. Transition control unit>

[0146] The transition control unit 21 is different from Figure 3 the transition control unit 15 shown in [the figure] in that it does not necessarily have to include functional units such as a return movement amount calculation unit 15B and a combined movement amount calculation unit 15C, which are compatible with the control of returning the viewpoint position to the recommended viewpoint. The transition control unit 21 is different from Figure 3 the transition control unit 15 shown in [the figure] also in that it includes a correction unit 21A that performs correction to limit the above-described operation movement amount.

[0147] <2-2-1. Correction unit>

[0148] The correction unit 21A is a processing unit that corrects the above-described operation movement amount based on the angular difference in the line-of-sight direction between the viewpoint position of the current frame and the recommended viewpoint.

[0149] Figure 11 and Figure 12 are graphs each showing an example method for correcting the operation movement amount. Figure 11 and Figure 12 In the graphs shown in [the figure], the horizontal axis indicates the angle relative to the recommended viewpoint, and the horizontal axis indicates the operation movement amount. Additionally, in Figure 11 and Figure 12 the corrected operation movement amount is indicated by a solid line, while the operation movement amount before correction is indicated by a dashed line. As shown in Figure 11 a function in which the rate of monotonic increase of the corrected operation movement amount is lower than the rate of monotonic increase of the operation movement amount before correction as the angle relative to the recommended viewpoint becomes larger can be used to correct the operation movement amount. As another example, when correcting the operation movement amount, a function in which the operation movement amount before correction and the corrected operation movement amount linearly increase with the same slope until the angle relative to the recommended viewpoint increases to a predetermined angle, and when the angle equivalent to the recommended viewpoint becomes equal to or greater than the predetermined angle, the slope of the linear increase of the corrected operation movement amount becomes gentler than the slope of the linear increase of the operation movement amount before correction can be used, as shown in Figure 12 the figure.

[0150] As described above, a function represented by log or a function represented by a threshold value is used as a function that restricts an increase in the operation movement amount as the angle from the recommended viewpoint becomes larger. With this arrangement, it is possible to transmit a state in which the viewpoint position is far from the recommended viewpoint or outside the range of the recommended viewpoint, where the restriction on the increase in the operation movement amount becomes a limiter. Thus, the recommended viewpoint can be visually feedback without displaying various marks such as borders and arrows indicating the position of the recommended viewpoint.

[0151] Note that, although Figure 11 and Figure 12 illustrate an example function for determining the operation movement amount before and after correction, the operation movement amount is not necessarily calculated according to the function. For example, instead of the function, a look-up table or the like can of course be used, in which the operation movement amount before and after correction is associated with the corresponding level of the rotation angle in the line-of-sight direction from the recommended viewpoint to the viewpoint position of the current frame.

[0152] <2-3. Processing procedure to be executed by the display device>

[0153] Figure 13 is a flowchart showing the process of the display process according to the second embodiment. In the Figure 13 flowchart shown, the same step numbers are assigned to the steps that execute the same content as the processing in the Figure 7 flowchart shown, but different step numbers are assigned to the steps that execute different content.

[0154] Figure 13 The display process shown in Figure 7 is different from the display process shown in Figure 7 in that the processing in steps S103 and S104 shown in Figure 7 is omitted, and the processing in step S301 is added. Specifically, only as an example, step S301 can be executed after the processing in step S102. For example, in step S301, the correction unit 21A corrects the operation movement amount calculated in step S102 based on the angular difference in the line-of-sight direction between the viewpoint position of the current frame and the recommended viewpoint.

[0155] Thereafter, based on the operation movement amount corrected in step S301, the subsequent processing in step S105 is executed.

[0156] <2-4. One aspect of the effect>

[0157] As described above, the display device 20 of the present disclosure corrects the above-mentioned operation movement amount based on the angular difference in the line-of-sight direction between the viewpoint position of the current frame and the recommended viewpoint. Thus, with the display device 20 of the present disclosure, the recommended viewpoint can be used in combination with the free viewpoint without necessarily indicating the position of the recommended viewpoint with a border or the like.

[0158] <<3. Third Embodiment>>

[0159] In the above example of the second embodiment, the guidance control for returning the viewpoint position to the recommended viewpoint is omitted from the above display device 10 according to the first embodiment, and the correction of the operation movement amount is incorporated into the display device 10. However, the above first embodiment and the above second embodiment can be combined without any change. In the example case described below as the third embodiment, the above first embodiment and the second embodiment are combined without any change.

[0160] <3-1. Example Functional Configuration of Display Device>

[0161] Figure 14 is a block diagram showing an example functional configuration of a display device according to the third embodiment. As Figure 14 shown, the display device 30 differs from the Figure 3 display device 10 shown in that it includes a transition control unit 31 that performs processing that is partially different in content from the processing performed by the transition control unit 15.

[0162] <3-2. Transition Control Unit>

[0163] The transition control unit 31 differs from the Figure 3 transition control unit 15 shown in that, in addition to the Figure 3 operation movement amount calculation unit 15A, return movement amount calculation unit 15B, combined movement amount calculation unit 15C, and viewpoint position calculation unit 15D shown in Figure 10 it further includes a correction unit 21A shown in

[0164] <3-3. Processing Procedure to be Executed by Display Device>

[0165] Figure 15 is a flowchart showing the process of the display processing according to the third embodiment. In the Figure 15 flowchart shown, the same step numbers are assigned to the steps that perform processing having the same content as the processing in the Figure 7 and Figure 13 flowcharts shown, but different step numbers are assigned to the steps that perform processing having different content.

[0166] Merely as an example, this processing also starts in response to a video clip reproduction start operation performed by the user. When the processing starts in this way, the display device 10 displays the recommended viewpoint as the display range within the range corresponding to the RVP (recommended range), as Figure 15 shown (step S101).

[0167] Thereafter, the subsequent processes in steps S102 to S104 are executed for each of the three axes of pitch, roll, and yaw.

[0168] That is, the operation movement amount calculation unit 15A calculates the above operation movement amount based on the line-of-sight operation information output by the gyro sensor 14 (step S102). Then, the correction unit 21A corrects the operation movement amount calculated in step S102 based on the angular difference between the line-of-sight direction of the viewpoint position in the current frame and the recommended viewpoint (step S301). In addition, the return movement amount calculation unit 15B calculates the above return movement amount based on the recommended viewpoint information output by the decoder 12 (step S103).

[0169] Thereafter, the combined movement amount calculation unit 15C calculates the combined movement amount by adding the operation movement amount corrected in step S301 and the return movement amount calculated in step S103 for each of the three axes of roll and yaw (step S104).

[0170] Subsequently, the viewpoint position calculation unit 15D calculates the viewpoint position of the next frame by adding the angle of the line-of-sight direction corresponding to the combined movement amount calculated for each of the three axes of roll and yaw in step S104 to the line-of-sight direction of the current frame (step S105).

[0171] Then, the field-of-view rendering unit 13 causes the display unit 16 to display a display image, and sets the field of view corresponding to the viewpoint position of the next frame calculated in step S105 as the display range of the projected image (step S106).

[0172] Thereafter, the processes in steps S102 to S106 are repeated until the video reproduction or distribution is completed ("No" in step S107). Thereafter, when the video reproduction or distribution is completed ("Yes" in step S107), the process ends.

[0173] <3-4. An aspect of the effect>

[0174] As described above, with the display device 30 of the present disclosure, the effects of both the display device 10 according to the first embodiment and the display device 20 according to the second embodiment can be achieved. For example, with the display device 30 of the present disclosure, the sense of distance to the recommended viewpoint, positions outside the range of the recommended viewpoint, etc. can be transmitted.

[0175] <<4. Fourth Embodiment>>

[0176] In the example cases in the first to third embodiments described above, there is one recommended view in the process. However, there may be multiple recommended viewpoints. In the example process described below as the fourth embodiment, one recommended viewpoint is selected from the multiple recommended viewpoints.

[0177] <4-1. Example functional configuration of the display device>

[0178] Figure 16 is a block diagram showing an example functional configuration of a display device according to the fourth embodiment. As Figure 16 shown, the display device 40 is Figure 3 different from the display device 10 shown in

[0179] <4-2. Transition control unit>

[0180] The transition control unit 41 is Figure 3 different from the transition control unit 15 shown in

[0181] <4-2-1. Fraction calculation unit>

[0182] The fraction calculation unit 41A is a processing unit that calculates the fraction of each recommended viewing point.

[0183] As an example only, the fraction calculation unit 41A can calculate the fraction of the recommended viewing point according to the distance between the viewing point position of the current frame and the recommended viewing point according to the following expression (1). "S" in the following expression (1) i " represents the fraction of the i-th recommended viewing point. At the same time, "d" in the following expression (1) i " represents the distance between the i-th recommended viewing point and the viewing point position of the current frame. Figure 17 is a diagram showing an example of the distance used in the fraction calculation. Figure 17 shows three recommended viewing points RVP1 to RVP3 and the viewing point position of the current frame in the projected image. In Figure 17 the example shown, the magnitude relationship among the distances d1, d2, and d3 between the three recommended viewing points RVP1 to RVP3 and the viewing point position of the current frame is "d1 < d3 < d2". Therefore, the magnitude relationship among the fractions of the recommended viewing points RVP1 to RVP3 is "S1 > S3 > S2".

[0184] S i = 1 / d i ...(1)

[0185] As another example, the fraction calculation unit 41A can calculate the fraction of the recommended viewing point by assigning a predetermined weight to the distance between the viewing point position of the current frame and the recommended viewing point according to the following expression (2). "S" in the following expression (2) i” represents the score of the i-th recommended viewpoint. Meanwhile, “d i " represents the distance between the i-th recommended viewpoint and the viewpoint position of the current frame. In addition, "w i " represents the weight of the i-th recommended viewpoint. For example, the weight of the recommended viewpoint can be set by a user of the distribution source such as a director, or by an end user who is a viewer. In addition, a larger weight can be automatically set for a recommended viewpoint with a large number of viewers by referring to the viewing history of the distribution data of multiple viewers.

[0186] S i =w i / d i ...(2)

[0187] Figure 18 is a diagram showing an example of distances and weights used in score calculation. Figure 17 Same, Figure 18 Three recommended viewpoints RVP1 to RVP3 and the viewpoint positions of the current frame in the projected image are shown. Figure 18 In , the weights of the recommended viewpoints RVP1 to RVP3 are represented by the size of the circles. Figure 18 In the example shown in FIG. 1 , the distances d1 , d2 , and d3 between the three recommended viewpoints RVP1 to RVP3 and the viewpoint position of the current frame are related by “d1 <d3<d2”。另一方面,三个推荐视点RVP1至RVP3的权重之间的大小关系为“w1=w2<w3”。在这种情况下,推荐视点RVP1至RVP3的分数之间的大小关系不一定是“S1> That is, depending on the weight w3 of the recommended viewpoint RVP3, in some cases, the size relationship between the scores of the recommended viewpoints RVP1 to RVP3 may be “S3>S1>S2”, such as Figure 18 as shown in .

[0188] <4-2-2. Recommended viewpoint selection unit>

[0189] The recommended viewpoint selection unit 41B is a processing unit that selects one recommended viewpoint from among a plurality of recommended viewpoints.

[0190] As just an example, the recommended viewpoint selection unit 41B selects a recommended viewpoint having the highest score among a plurality of recommended viewpoints. Figure 17 In the example shown in , the recommended viewpoint RVP1 is selected from among the recommended viewpoints RVP1 to RVP3, and Figure 18 In the example shown in , the recommended viewpoint RVP3 is selected from among the recommended viewpoints RVP1 to RVP3.

[0191] By selecting the recommended viewpoint with the highest score in this way, the nearest recommended viewpoint or the most view-worthy recommended viewpoint in the recommended distribution data can be recommended.

[0192] Note that in the examples described herein, the score with a higher priority is calculated from the recommended viewpoints with higher score values. However, the present disclosure is not limited to this example. For example, the score with a higher priority can be calculated from the recommended viewpoints with lower score values. In this case, it is only necessary to select the recommended viewpoint with the lowest score.

[0193] <4-3. Processing procedures to be executed by the display device>

[0194] Figure 19 is a flowchart showing the process of the display process according to the fourth embodiment. In Figure 19 In the flowchart shown, the same step numbers are assigned to the steps that execute the same content as the processing in the flowchart shown in Figure 7 but different step numbers are assigned to the steps that execute different content.

[0195] Figure 19 The display process shown in Figure 7 is different from the display process shown in Figure 7 in that the processing in steps S401 and S402 is executed after the processing in step S101 shown in

[0196] Specifically, in step S401, as an example only, the score calculation unit 41A calculates the score for each recommended viewpoint. After that, the recommended viewpoint selection unit 41B selects one recommended viewpoint from the multiple recommended viewpoints based on the score calculated in step S401 (step S402).

[0197] <4-4. One aspect of the effect>

[0198] As described above, the display device 40 of the present disclosure calculates the score for each recommended viewpoint and selects one recommended viewpoint from the multiple recommended viewpoints based on the score. Therefore, with the display device 40 of the present disclosure, one recommended viewpoint can be selected even when there are multiple recommended viewpoints.

[0199] <<5. Modification examples>>

[0200] In the following description, modification examples of the first to fourth embodiments are described.

[0201] <5-1. Non-visual feedback>

[0202] Although an example of transmitting the position of the recommended viewing point by changing the display range of the projected image has been described in the first to fourth embodiments, the position of the recommended viewing point may be transmitted by a method other than display. For example, the position of the recommended viewing point may be transmitted by audio output. As an example, the volume may be decreased as the distance from the recommended viewing point increases, or the sound source in the direction of the recommended viewing point may be made perceptible to the user by positioning the sound image at the recommended viewing point through stereo. In addition to such sound output, when the distance from the recommended viewing point to the viewing point position of the current frame or the next frame increases by an amount equal to or greater than a predetermined value, the user may be made to perceive vibration by activating a vibrator or the like, or the user may perceive a heavier weight through a gyroscope as the distance from the recommended viewing point increases.

[0203] <5-2. Execution Conditions of Guidance Control>

[0204] In the examples described in the first, third, and fourth embodiments, guidance control for returning the viewing point position to the recommended viewing point is executed for each frame. However, specific conditions may also be set. For example, the display devices 10, 30, or 40 may execute the guidance control for returning the viewing point position to the recommended viewing point only when the operation movement amount is zero or equal to or less than a predetermined threshold, or when there is no line-of-sight operation. For example, when the operation movement amount is zero or equal to or less than a predetermined threshold, the return movement amount calculation unit may calculate the return movement amount.

[0205] <5-3. Devices for Implementing Functions>

[0206] In the examples described in the first to fourth embodiments, the transition control units 15, 21, 31, and 41 are installed in the display devices 10, 20, 30, and 40. However, the transition control units 15, 21, 31, and 41 may be installed in the distribution server 9. In this case, it is only necessary to obtain the line-of-sight operation information generated by the gyro sensor 14 from the display devices 10, 20, 30, or 40. As described above, in addition to the display devices 10, 20, 30, and 40, the distribution server 9 may also correspond to an example of an image processing device.

[0207] <5-4. Other Variations>

[0208] In addition, among the various processes described in the above embodiments, all or part of the processes described as being automatically executed may be manually executed, or all or part of the processes described as being manually executed may be automatically executed by known methods. In addition to this, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters mentioned above in the document and shown in the drawings may be appropriately changed. For example, the various information shown in the corresponding drawings is not limited to the information shown in the drawings.

[0209] In addition, each component of each device shown in the drawings is conceptual in function and does not need to be physically formed as shown in the drawings. That is, the specific distribution and integration mode of each device are not limited to the mode shown in the drawings, and according to various loads, usage conditions, etc., all or part of them can be functionally or physically distributed and integrated by appropriate units.

[0210] Furthermore, the beneficial effects of the corresponding embodiments described in this specification are merely examples, and the beneficial effects of the present technology are not limited to them and may include other effects.

[0211] <<6. Hardware Configuration>>

[0212] The serial processing in the above display devices 10, 20, 30, or 40 can be executed by hardware or can also be executed by software. In the case where the serial processing is to be executed by software, the program forming the software is installed in a computer. Here, the computer can be a computer incorporated into dedicated hardware or can be, for example, a general-purpose personal computer that can execute various functions and in which various programs are installed.

[0213] Figure 20 is a block diagram showing an example configuration of the hardware of the computer 400.

[0214] In the computer 400, a central processing unit (CPU) 401, a read-only memory (ROM) 402, and a random access memory (RAM) 403 are interconnected via a bus 404.

[0215] An input / output interface 405 is further connected to the bus 404. An input unit 406, an output unit 407, a recording unit 408, a communication unit 409, and a driver 410 are connected to the input / output interface 405.

[0216] The input unit 406 is composed of an input switch, a button, a microphone, an imaging element, etc. The output unit 407 is composed of a display, a speaker, etc. The recording unit 408 is composed of a hard disk, a non-volatile memory, etc. The communication unit 409 is composed of a network interface, etc. The driver 410 drives a removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0217] In the computer 400 having the above configuration, the CPU 401 loads, for example, the program recorded in the recording unit 408 into the RAM 403 via the input / output interface 405 and the bus 404 and executes the program, thereby performing the above serial processing.

[0218] For example, a program to be executed by a computer 400 (CPU 401) can be recorded on a removable medium 411 which is a packaged medium to be provided. Alternatively, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0219] In a computer, when the removable medium 411 is mounted on a drive 410, the program can be installed into a recording unit 408 via an input / output interface 405. The program can also be received by a communication unit 409 via a wired or wireless transmission medium and installed into the recording unit 408. Alternatively, the program can be pre-installed into a ROM 402 or the recording unit 408.

[0220] Note that a program to be executed by a computer can be a program for executing processing in chronological order according to the order described in this specification, or can be a program for executing processing in parallel or when necessary (such as when there is a phone call).

[0221] Note that this technology can also be implemented in the configurations described below. (1)

[0223] An image processing apparatus, comprising:

[0224] An output unit that outputs a part of an image including recommended viewpoint information as a display image to a display unit; and

[0225] A transformation control unit that changes a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, where

[0226] The output unit outputs a part of the image to the display unit based on the changed display range. (2)

[0228] The image processing apparatus according to (1), wherein

[0229] The transformation control unit includes:

[0230] An operation movement amount calculation unit that calculates an operation movement amount for moving a viewpoint position corresponding to the display image based on a viewpoint operation according to viewpoint operation information;

[0231] A return movement amount calculation unit that calculates a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and

[0232] A viewpoint position calculation unit that calculates a viewpoint position of a next frame to be output after a frame of the display image output to the display unit based on the operation movement amount and the return movement amount, and

[0233] Output unit

[0234] Output a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame. (3)

[0236] The image processing apparatus according to (2), wherein

[0237] The return movement amount calculation unit calculates the return movement amount based on the angular difference in the line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the displayed image. (4)

[0239] The image processing apparatus according to (3), wherein

[0240] As the angular difference in the line-of-sight direction becomes larger, the return movement amount calculation unit sets the speed of returning to the viewpoint position corresponding to the recommended viewpoint information to be higher. (5)

[0242] The image processing apparatus according to (4), wherein

[0243] As the angular difference in the line-of-sight direction becomes larger, the return movement amount calculation unit linearly increases the speed of returning to the viewpoint position corresponding to the recommended viewpoint information. (6)

[0245] The image processing apparatus according to (4), wherein

[0246] As the angular difference in the line-of-sight direction becomes larger, the return movement amount calculation unit monotonically increases the speed of returning to the viewpoint position corresponding to the recommended viewpoint information and reduces the ratio of the monotonic increase. (7)

[0248] The image processing apparatus according to any one of (2) to (6), wherein

[0249] The viewpoint position calculation unit calculates the viewpoint position of the next frame based on the return movement amount when the operation movement amount is equal to or less than a predetermined threshold, and calculates the viewpoint position of the next frame based on the operation movement amount and the return movement amount when the operation movement amount is neither equal to nor less than the threshold. (8)

[0251] The image processing apparatus according to (7), wherein

[0252] The value of the threshold becomes larger as the angular difference in the line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the displayed image becomes smaller. (9)

[0254] The image processing apparatus according to any one of (1) to (8), wherein

[0255] The transformation control unit includes:

[0256] An operation movement amount calculation unit that calculates an operation movement amount for moving a viewpoint position corresponding to a displayed image based on a viewpoint operation according to viewpoint operation information;

[0257] A correction unit that corrects the operation movement amount based on an angular difference in the line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the displayed image; and

[0258] A viewpoint position calculation unit that calculates a viewpoint position of a next frame to be output after a frame of the displayed image output to the display unit based on the corrected operation movement amount, and

[0259] An output unit

[0260] Outputs a part of the image to the display unit based on a display range determined from the viewpoint position of the next frame. (10)

[0262] The image processing apparatus according to (9), wherein

[0263] As the angular difference in the line-of-sight direction becomes larger, the correction unit makes the monotonic increase ratio of the corrected operation movement amount lower than the monotonic increase ratio of the operation movement amount before correction. (11)

[0265] The image processing apparatus according to (9), wherein

[0266] The correction unit linearly increases the operation movement amount before correction and the corrected operation movement amount at the same slope until the angular difference in the line-of-sight direction increases to a predetermined angle, and when the angular difference in the line-of-sight direction becomes equal to or greater than the predetermined angle, the correction unit linearly increases the corrected operation movement amount at a slope gentler than the slope at which the correction unit linearly increases the operation movement amount before correction. (12)

[0268] The image processing apparatus according to any one of (2) to (11) further includes:

[0269] A score calculation unit that calculates a score for each viewpoint position included in the recommended viewpoint information; and

[0270] A recommended viewpoint selection unit that selects one of the viewpoint positions included in the recommended viewpoint information based on the score, wherein

[0271] The return movement amount calculation unit calculates a return movement amount for returning the viewpoint position corresponding to the display image to a viewpoint position selected from among the viewpoint positions included in the recommended viewpoint information. (13)

[0273] The image processing apparatus according to (12), wherein

[0274] The score calculation unit calculates a score based on the distance between the viewpoint position included in the recommended viewpoint information and the viewpoint position corresponding to the display image. (14)

[0276] The image processing apparatus according to (13), wherein

[0277] As the distance becomes longer, the score calculation unit makes the value of the score smaller. (15)

[0279] The image processing apparatus according to (13), wherein

[0280] The score calculation unit calculates a score by assigning a predetermined weight to the distance. (16)

[0282] The image processing apparatus according to (15), wherein

[0283] The score calculation unit makes the value of the score larger as the weight becomes larger, or makes the value of the score smaller as the weight becomes smaller. (17)

[0285] The image processing apparatus according to (16), wherein

[0286] A larger value is assigned to the weight of the recommended viewpoint having a larger number of viewers. (18)

[0288] The image processing apparatus according to any one of (2) to (17), wherein

[0289] When the operation movement amount is equal to or less than a predetermined threshold, the return movement amount calculation unit calculates the return movement amount. (19)

[0291] An image processing method executed by a computer, the method performing the following processing:

[0292] Output a part of an image including recommended viewpoint information as a display image to a display unit; and

[0293] Change the display range of the image based on the positional relationship between the viewpoint position corresponding to the display image output to the display unit and the viewpoint position corresponding to the recommended viewpoint information, wherein

[0294] The process of outputting a part of an image to a display unit includes outputting a part of the image to the display unit based on the changed display range. (20)

[0296] A program for causing a computer to execute the following processes:

[0297] Outputting a part of an image including recommended viewpoint information as a display image to a display unit; and

[0298] Changing the display range of the image based on the positional relationship between the viewpoint position corresponding to the display image output to the display unit and the viewpoint position corresponding to the recommended viewpoint information, where

[0299] The process of outputting a part of an image to a display unit includes outputting a part of the image to the display unit based on the changed display range.

[0300] List of reference signs

[0301] 1 Image distribution system

[0302] 3 Multi-camera

[0303] 5 Wide-angle image conversion unit

[0304] 7 Encoder

[0305] 9 Distribution server

[0306] 10 Display device

[0307] 11 Receiving unit

[0308] 12 Decoder

[0309] 13 Field-of-view rendering unit

[0310] 14 Gyro sensor

[0311] 15 Transition control unit

[0312] 15A Operation movement amount calculation unit

[0313] 15B Return movement amount calculation unit

[0314] 15C Combined movement amount calculation unit

[0315] 15D Viewpoint position calculation unit

[0316] 16 Display unit

Claims

1. An image processing apparatus, comprising: an output unit that outputs a part of an image including recommended viewpoint information as a display image to a display unit; and a transformation control unit that changes a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, wherein the output unit outputs a part of the image to the display unit based on the changed display range, wherein the transformation control unit includes: an operation movement amount calculation unit that calculates an operation movement amount for moving a viewpoint position corresponding to the display image based on a viewpoint operation according to viewpoint operation information; a return movement amount calculation unit that calculates a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and a viewpoint position calculation unit that calculates a viewpoint position of a next frame to be output after a frame of the display image output to the display unit based on the operation movement amount and the return movement amount, wherein the output unit outputs a part of the image to the display unit based on a display range determined from the viewpoint position of the next frame.

2. The image processing apparatus according to claim 1, wherein the return movement amount calculation unit calculates the return movement amount based on an angular difference in a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image.

3. The image processing apparatus according to claim 2, wherein as the angular difference in the line-of-sight direction becomes larger, the return movement amount calculation unit sets a speed for returning to the viewpoint position corresponding to the recommended viewpoint information to be higher.

4. The image processing apparatus according to claim 3, wherein as the angular difference in the line-of-sight direction becomes larger, the return movement amount calculation unit linearly increases the speed for returning to the viewpoint position corresponding to the recommended viewpoint information.

5. The image processing apparatus according to claim 3, wherein as the angular difference in the line-of-sight direction becomes larger, the return movement amount calculation unit monotonically increases the speed for returning to the viewpoint position corresponding to the recommended viewpoint information and reduces a ratio of the monotonic increase.

6. The image processing apparatus according to claim 1, wherein the viewpoint position calculation unit calculates the viewpoint position of the next frame based on the return movement amount when the operation movement amount is equal to or less than a predetermined threshold, and calculates the viewpoint position of the next frame based on the operation movement amount and the return movement amount when the operation movement amount is neither equal to nor less than the threshold.

7. The image processing apparatus according to claim 6, wherein a value of the threshold becomes larger as an angular difference in a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image becomes smaller.

8. The image processing apparatus according to claim 1, wherein the transformation control unit further includes: a correction unit that corrects the operation movement amount based on an angular difference in a line-of-sight direction from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image, wherein the viewpoint position calculation unit calculates the viewpoint position of the next frame to be output after a frame of the display image output to the display unit based on the corrected operation movement amount.

9. The image processing apparatus according to claim 8, wherein As the angular difference in the line-of-sight direction increases, the correction unit causes the ratio of the monotonically increasing corrected operation movement amount to be lower than the ratio of the monotonically increasing operation movement amount before correction.

10. The image processing apparatus according to claim 8, wherein The correction unit linearly increases the operation movement amount before correction and the corrected operation movement amount with the same slope until the angular difference in the line-of-sight direction increases to a predetermined angle. And when the angular difference in the line-of-sight direction becomes equal to or greater than the predetermined angle, the correction unit linearly increases the corrected operation movement amount with a slope gentler than the slope of the linear increase of the operation movement amount before correction.

11. The image processing apparatus according to claim 1, further comprising: A score calculation unit that calculates a score for each viewpoint position included in the recommended viewpoint information; And A recommended viewpoint selection unit that selects one of the viewpoint positions included in the recommended viewpoint information based on the score, wherein The return movement amount calculation unit calculates a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position selected from among the viewpoint positions included in the recommended viewpoint information.

12. The image processing apparatus according to claim 11, wherein The score calculation unit calculates the score based on the distance between the viewpoint position included in the recommended viewpoint information and the viewpoint position corresponding to the display image.

13. The image processing apparatus according to claim 12, wherein As the distance becomes longer, the score calculation unit makes the value of the score smaller.

14. The image processing apparatus according to claim 12, wherein The score calculation unit calculates the score by assigning a predetermined weight to the distance.

15. The image processing apparatus according to claim 14, wherein The score calculation unit makes the value of the score increase as the weight increases, or makes the value of the score decrease as the weight decreases.

16. The image processing apparatus according to claim 15, wherein A larger value is assigned to the weight of the recommended viewpoint with a larger number of viewers.

17. The image processing apparatus according to claim 1, wherein When the operation movement amount is equal to or less than a predetermined threshold, the return movement amount calculation unit calculates the return movement amount.

18. An image processing method executed by a computer, the method performing the following processing: Outputting a part of an image including recommended viewpoint information as a display image to a display unit; and Changing the display range of the image based on the positional relationship between the viewpoint position corresponding to the display image output to the display unit and the viewpoint position corresponding to the recommended viewpoint information, The process of outputting a part of an image to a display unit includes: Based on the changed display range, outputting a part of the image to the display unit, Wherein the processing of changing the display range of the image further includes: Calculating an operation movement amount for moving the viewpoint position corresponding to the display image according to a viewpoint operation based on viewpoint operation information; Calculating a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and Calculating the viewpoint position of the next frame to be output after the frame of the display image output to the display unit based on the operation movement amount and the return movement amount, The process of outputting a part of the image to the display unit includes: outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame.

19. A program product including instructions for causing a computer to perform the following processes: Outputting a part of the image including the recommended viewpoint information as a display image to the display unit; and Changing the display range of the image based on the positional relationship between the viewpoint position corresponding to the display image output to the display unit and the viewpoint position corresponding to the recommended viewpoint information, The process of outputting a part of the image to the display unit includes: Outputting a part of the image to the display unit based on the changed display range, wherein the process of changing the display range of the image further includes: Calculating an operation movement amount for moving the viewpoint position corresponding to the display image according to a viewpoint operation based on the viewpoint operation information; Calculating a return movement amount for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information; and Calculating the viewpoint position of the next frame to be output after the frame of the display image output to the display unit based on the operation movement amount and the return movement amount, wherein the process of outputting a part of the image to the display unit includes: outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame.

Images