Image processing device, image processing method and program

Abstract

A display device (10) which is one example of this image processing device comprises an output unit that outputs a portion of an image including recommended viewpoint information to a display unit as a display image, and a transition control unit that transitions the display region of the image on the basis of a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information. The output unit outputs a portion of the image to the display unit on the basis of the transitioned display region.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] The standard of OMAF (Omnidirectional Media Application Format) of MPEG-DASH (Dynamic Adaptive Streaming over HTTP) defines a method of sending recommended viewpoint information together with image data of a wide viewing angle of 360 degrees or narrower than that. In addition, for example, Patent Document 1 has a description regarding display of a recommended viewpoint. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-194784 A Summary of the Invention [Problem to be solved by the invention]

[0004] If the recommended viewpoint is displayed, only the recommended viewpoint can be seen in the wide-angle image. On the other hand, if the viewer can freely view the image, the recommended viewpoint position becomes unclear. In this case, it is possible to display the recommended viewpoint with a frame or the like, but this has the disadvantage of destroying the view of the world in that area.

[0005] Therefore, an object of the present disclosure is to provide an image processing device, an image processing method, and a program that enable the combined use of a recommended viewpoint and a free viewpoint without displaying the position of the recommended viewpoint with a frame or the like. [Means for solving the problem]

[0006] In order to solve the above problems, an image processing device of one embodiment according to the present disclosure includes an output unit that outputs a portion of an image including recommended viewpoint information to a display unit as a display image, and a transition control unit that transitions a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, and the output unit outputs a portion of the image to the display unit based on the transitioned display range. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an image delivery system according to the first embodiment. [Figure 2A] FIG. 2A is a diagram showing an example of a sphere corresponding to a spherically captured image. [Figure 2B] FIG. 2B is a schematic diagram showing an example of a coordinate system of a projection image. [Figure 2C] FIG. 2C is a diagram showing an example of recommended viewpoint information. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the display device according to the first embodiment. [Figure 4A] FIG. 4A is a diagram showing an example of a projection image. [Figure 4B] FIG. 4B is a diagram showing an example of a display range. [Figure 4C] FIG. 4C is a diagram showing an example of a display image. [Diagram 5] FIG. 5 is a diagram illustrating an example of a method for calculating the return movement amount. [Figure 6] FIG. 6 is a diagram illustrating an example of a method for calculating the return movement amount. [Figure 7] FIG. 7 is a flowchart showing a procedure of the display process according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a method for setting a threshold value. [Figure 9] FIG. 9 is a flowchart showing a procedure of a display process according to an application example of the first embodiment. [Figure 10]FIG. 10 is a block diagram illustrating an example of a functional configuration of the display device according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a method for correcting the operation movement amount. [Figure 12] FIG. 12 is a diagram illustrating an example of a method for correcting the operation movement amount. [Figure 13] FIG. 13 is a flowchart showing the procedure of the display process according to the second embodiment. [Figure 14] FIG. 14 is a block diagram illustrating an example of a functional configuration of the display device according to the third embodiment. [Figure 15] FIG. 15 is a flowchart showing the procedure of the display process according to the third embodiment. [Figure 16] FIG. 16 is a block diagram illustrating an example of a functional configuration of a display device according to the fourth embodiment. [Figure 17] FIG. 17 is a diagram showing an example of a distance used in score calculation. [Figure 18] FIG. 18 is a diagram showing an example of distances and weights used in score calculation. [Figure 19] FIG. 19 is a flowchart showing the procedure of the display process according to the fourth embodiment. [Figure 20] FIG. 20 is a block diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0009] The present disclosure will be described in the following order. 1. First embodiment 1-1. Image distribution system 1-2. Distribution device 1-3. Receiving device 1-3-1. Receiving section 1-3-2.Gyro sensor 1-3-3.Transition control section 1-3-4. Visual field drawing section 1-3-5.Display section 1-4.Details of the transition control section 1-4-1. Operation movement amount calculation section 1-4-2.Return movement amount calculation section 1-4-3.Composite movement amount calculation section 1-4-4. Viewpoint position calculation section 1-5. Display device processing procedure 1-6. One aspect of the effect 1-7. Application example of the first embodiment 1-7-1. Application example processing procedure 2. Second embodiment 2-1. Example of display device function configuration 2-2.Transition control section 2-2-1. Correction section 2-3. Display device processing procedure 2-4. One aspect of the effect 3. Third embodiment 3-1. Example of display device function configuration 3-2.Transition control section 3-3. Display device processing procedure 3-4. One aspect of the effect 4. Fourth embodiment 4-1. Example of display device function configuration 4-2.Transition control section 4-2-1.Score calculation section 4-2-2. Recommended viewpoint selection section 4-3. Display device processing procedure 4-4. One aspect of the effect 5. Variations 5-1. Non-visual feedback 5-2. Guidance control execution conditions 5-3. Function implementation subject 5-4.Other modifications 6. Hardware Configuration

[0010] <<1. First embodiment>> <1-1. Image distribution system> Fig. 1 is a diagram showing a configuration example of an image delivery system according to a first embodiment. As shown in Fig. 1, the image delivery system 1 has a delivery device and a display device 10 as a receiving device. Examples of the display device 10 include an HMD (Head Mounted Display) and a tablet.

[0011] <1-2. Distribution device> The distribution device includes a multi-camera 3, a wide viewing angle image conversion unit 5, an encoder 7, and a distribution server 9.

[0012] The multi-camera 3 obtains image data of a spherical capture image. For example, the multi-camera 3 performs imaging using a back-to-back method with two cameras to obtain a wide-angle front image and a wide-angle rear image, each of which has a viewing angle of 180° or more and is captured using a fisheye lens, as a spherical capture image.

[0013] The wide viewing angle image conversion unit 5 performs plane packing of the spherical capture image obtained by the multi-camera 3 to obtain a rectangular projection image. This projection image corresponds to an image with a wide viewing angle of 360° or a narrower angle of view. In this case, for example, equirectangular is selected as the format type of the projection image. In addition, the wide viewing angle image conversion unit 5 performs scaling on the projection image as necessary to obtain a projection image with a predetermined resolution.

[0014] The encoder 7 obtains coded image data by, for example, HEVC coding on the image data of the projection image from the wide viewing angle image converter 5, and generates a video stream including the coded image data. Here, the encoder 7 inserts recommended viewpoint information into the video stream on a frame-by-frame basis.

[0015] The recommended viewpoint (RVP) is automatically set to a range including the performer estimated by a position sensor or image recognition, or is set to a range manually specified by the director, for example. Here, the recommended viewpoint is not necessarily set to only one, and multiple recommended viewpoints may be set.

[0016] Figure 2A shows a sphere corresponding to a spherical captured image. Figure 2B shows a schematic of a rectangular projected image when the format type is equirectangular. The center of the projected image is (0,0).

[0017] The recommended viewpoint information includes, for example, a frame number, central angle information, horizontal angle information, vertical angle information, a recommended viewpoint number, etc., as shown in Fig. 2C. The central angle information, horizontal angle information, and vertical angle information are angle information in a spherical capture image (see Fig. 2A).

[0018] Returning to Fig. 1, the distribution server 9 transmits the video stream generated by the encoder 7 as distribution data to the display device 10. Note that this distribution may be by either broadcasting or communication.

[0019] <1-3. Receiving device> Fig. 3 is a block diagram showing an example of a functional configuration of the display device according to the first embodiment. In Fig. 3, blocks corresponding to functions of the display device 10 are diagrammed. As shown in Fig. 3, the display device 10 has a receiving unit 11, a decoder 12, a visual field drawing unit 13, a gyro sensor 14, a transition control unit 15, and a display unit 16. The visual field drawing unit 13 corresponds to an example of an output unit.

[0020] <1-3-1. Receiving section> The receiving unit 11 receives a video stream as distribution data sent from a transmitting device. The decoder 12 obtains a projection image (image data) by decoding the video stream received by the receiving unit 11. The decoder 12 also obtains recommended viewpoint information inserted on a frame-by-frame basis from the received video stream.

[0021] <1-3-2. Gyro sensor> The gyro sensor 14 detects a change in the rotation angle of a device incorporating the gyro sensor 14, in this case, the display device 10. Such a detection output may correspond to gaze operation information, such as a change in the rotation angle of the display device 10 when a user wearing an HMD turns his / her head or when a user turns a tablet. Note that this may also be applied to a use case in which a viewpoint operation occurs when a user performs an operation such as a swipe on a touch panel. In this case, the operation information of the touch panel may correspond to viewpoint operation information.

[0022] <1-3-3. Transition control section> The transition control unit 15 transitions the display range of the display image to be displayed on the display unit 16 among the projection images, based on the recommended viewpoint information for the next frame from the decoder 12 and the viewpoint operation information from the gyro sensor 14. Such transition of the display range can be realized, for example, by calculating the viewpoint position of the next frame for each frame. For example, the transition control unit 15 can determine, as the display range of the next frame, a field of view determined according to the horizontal angle of view and the vertical angle of view or the diagonal angle of view set as the size of the display image, with the viewpoint position of the next frame as the optical center.

[0023] <1-3-4. Visual Field Drawing Section> The field of view rendering unit 13 extracts and renders image data of the display range of the next frame calculated by the transition control unit 15 from the projection image data of the next frame obtained by the decoder 12, thereby obtaining display image data corresponding to the viewpoint position of the next frame.

[0024] <1-3-5.Display section> The display unit 16 displays an image based on the display image data obtained by the visual field rendering unit 13 for each frame.

[0025] FIG. 4A is a diagram showing an example of a projection image. FIG. 4B is a diagram showing an example of a display range. FIG. 4C is a diagram showing an example of a display image. As shown in FIG. 4B, a display range of the next frame corresponding to the viewpoint position of the next frame is set on the projection image shown in FIG. 4A, as an example only. When such a display range is set, a display image corresponding to the display range is cut out from the projection image and then displayed on the display unit 16, as shown in FIG. 4C.

[0026] <1-4. Details of the transition control section> As shown in FIG. 3, the transition control unit 15 has an operation movement amount calculation unit 15A, a return movement amount calculation unit 15B, a composite movement amount calculation unit 15C, and a viewpoint position calculation unit 15D.

[0027] <1-4-1. Operation movement amount calculation section> The operation movement amount calculation unit 15A is a processing unit that calculates the movement amount by which the viewpoint position corresponding to the display image is moved according to the viewpoint operation based on the viewpoint operation information. Hereinafter, the movement amount by which the viewpoint position corresponding to the display image is moved according to the viewpoint operation may be referred to as the "operation movement amount."

[0028] As an embodiment, the operation movement amount calculation unit 15A calculates the above operation movement amount based on the gaze operation information output by the gyro sensor 14. More specifically, the operation movement amount calculation unit 15A receives the angular velocities of the three axes, pitch, roll, and yaw, from the gyro sensor 14 on a frame-by-frame basis. In accordance with such input, the operation movement amount calculation unit 15A performs the following process for each of the three axes. That is, the operation movement amount calculation unit 15A calculates the rotation angle by integrating the angular velocities output by the gyro sensor 14. In this way, the operation movement amount calculation unit 15A calculates the operation movement amount for the next frame by subtracting the rotation angle calculated in the previous frame from the rotation angle calculated in the current frame.

[0029] <1-4-2. Return movement amount calculation section> The return movement amount calculation unit 15B is a processing unit that calculates the amount of movement for returning the viewpoint position corresponding to the display image to the viewpoint position corresponding to the recommended viewpoint information. Hereinafter, the amount of movement for returning the viewpoint position corresponding to the display image to the recommended viewpoint may be referred to as the "return movement amount."

[0030] As an embodiment, the return movement amount calculation unit 15B calculates the return movement amount based on the recommended viewpoint information output by the decoder 12. More specifically, the return movement amount calculation unit 15B also performs the following process for each of the three axes, pitch, roll, and yaw. That is, the return movement amount calculation unit 15B calculates the return movement amount based on the angle difference in the line of sight from the recommended viewpoint to the viewpoint position of the current frame. For example, the return movement amount calculation unit 15B sets a higher speed for returning to the recommended viewpoint as the angle difference in the line of sight from the recommended viewpoint to the viewpoint position of the current frame increases.

[0031] 5 and 6 are diagrams showing an example of a method for calculating the amount of return movement. The horizontal axis of the graphs shown in FIG. 5 and FIG. 6 indicates the angle from the recommended viewpoint, and the horizontal axis indicates the speed of returning to the recommended viewpoint. The "speed" referred to here is merely an example, and indicates the amount of movement per frame, and can be determined by setting the number of frames until returning to the recommended viewpoint. As shown in FIG. 5, the amount of return movement can be calculated according to a function in which the speed of returning to the recommended viewpoint increases linearly as the angle difference in the line of sight from the recommended viewpoint to the viewpoint position of the current frame increases. As another example, the amount of return movement can also be calculated according to a function in which the speed of returning to the recommended viewpoint increases monotonically and the rate of monotonous increase decreases as the angle difference in the line of sight from the recommended viewpoint to the viewpoint position of the current frame increases.

[0032] In this way, by incorporating the guidance control for returning the viewpoint position to the recommended viewpoint into the transition of the display range, the recommended viewpoint can be visually fed back without displaying various signs such as frames and arrows indicating the location of the recommended viewpoint. Furthermore, by calculating the amount of return movement described above according to the functions shown in Fig. 5 and Fig. 6, the sense of distance to the recommended viewpoint can be conveyed, thereby reducing VR sickness and reducing the sense of discomfort when guiding to the recommended viewpoint. Furthermore, by calculating the amount of return movement described above according to the functions shown in Fig. 6, the sense of distance to the recommended viewpoint can be conveyed more smoothly than in the case shown in Fig. 5.

[0033] Although an example in which the same function is used for the three axes of pitch, roll, and yaw has been given here, different functions may be used for each of the three axes. For example, for the roll axis, the return movement amount may be set to zero regardless of the magnitude of the angle difference in the line of sight from the recommended viewpoint to the viewpoint position of the current frame. In addition, functions multiplied by different weights may be used for each of the three axes.

[0034] 5 and 6 show examples of functions that determine the amount of return movement, but the amount of return movement does not necessarily have to be calculated according to a function. For example, it goes without saying that a lookup table or the like in which the speed of returning to the recommended viewpoint is associated with each class of the rotation angle of the line of sight from the recommended viewpoint to the viewpoint position of the current frame can be used instead of a function.

[0035] <1-4-3. Composite movement amount calculation section> The composite movement amount calculation unit 15C is a processing unit that calculates a composite movement amount obtained by combining the operation movement amount and the return movement amount. As an example, the composite movement amount calculation unit 15C sums up the operation movement amount calculated by the operation movement amount calculation unit 15A and the return movement amount calculated by the return movement amount calculation unit 15B for each of the three axes of roll and yaw. This makes it possible to calculate a composite movement amount obtained by combining the operation movement amount and the return movement amount.

[0036] <1-4-4. Viewpoint position calculation section> The viewpoint position calculation unit 15D is a processing unit that calculates the viewpoint position of the next frame from the composite movement amount. As an example, the viewpoint position calculation unit 15D calculates the viewpoint position of the next frame by adding the angle of the line of sight corresponding to the composite movement amount calculated for each of the three axes of roll and yaw by the composite movement amount calculation unit 15C to the line of sight direction of the current frame. Then, the viewpoint position calculation unit 15D determines the field of view determined according to the horizontal angle of view and the vertical angle of view or the diagonal angle of view set as the size of the display image, with the viewpoint position of the next frame as the optical center, as the display range of the next frame. The display range of the next frame determined in this way is input to the field of view drawing unit 13.

[0037] <1-5. Display device processing procedure> Fig. 7 is a flowchart showing the procedure of the display process according to the first embodiment. This process is started in response to a user's operation to start playing a video clip, as an example. When the process is started in this manner, the display device 10 displays the recommended viewpoint, that is, the range (recommended range) corresponding to the RVP, as the display range, as shown in Fig. 7 (step S101).

[0038] Thereafter, the processes from step S102 to step S104 described below are performed for each of the three axes, pitch, roll, and yaw.

[0039] That is, the operation movement amount calculation unit 15A calculates the above operation movement amount based on the line-of-sight operation information output by the gyro sensor 14 (step S102). Also, the return movement amount calculation unit 15B calculates the above return movement amount based on the recommended viewpoint information output by the decoder 12 (step S103).

[0040] Then, the composite movement amount calculation unit 15C calculates a composite movement amount by summing up the operation movement amount calculated in step S102 and the return movement amount calculated in step S103 for each of the three axes of roll and yaw (step S104).

[0041] Next, viewpoint position calculation unit 15D calculates the viewpoint position of the next frame by adding the line of sight direction angle corresponding to the composite movement amount calculated for each of the three axes of roll and yaw in step S104 to the line of sight direction of the current frame (step S105).

[0042] Then, the visual field rendering unit 13 causes the display unit 16 to display the display image with the visual field corresponding to the viewpoint position of the next frame calculated in step S105 as the display range of the projection image (step S106).

[0043] Thereafter, the processes from the above step S102 to the above step S106 are repeated until the video playback or distribution is ended (step S107 No). When the video playback or distribution is ended (step S107 Yes), the process is ended.

[0044] <1-6. One aspect of the effect> As described above, the display device 10 of the present disclosure transitions the display range of the projection image based on the line-of-sight operation information and the recommended viewpoint information, and displays a display image corresponding to the transitioned display range on the display unit 16. Therefore, according to the display device 10 of the present disclosure, it is possible to use the recommended viewpoint and the free viewpoint in combination, without displaying the position of the recommended viewpoint with a frame or the like.

[0045] <1-7. Application example of the first embodiment> In order to suppress vibrations near the recommended viewpoint, the display device 10 of the present disclosure can perform threshold processing to determine that there is no gaze operation when the amount of operation movement is equal to or less than a predetermined threshold. FIG. 8 is a diagram showing an example of a method for setting a threshold. The horizontal axis of the graph shown in FIG. 8 indicates the angle from the recommended viewpoint, and the vertical axis indicates the threshold for determining that there is no gaze operation. As shown in FIG. 8, the smaller the angle difference in the gaze direction from the recommended viewpoint to the viewpoint position of the current frame, the higher the threshold for determining that there is no gaze operation is set. In other words, the larger the angle difference in the gaze direction from the recommended viewpoint to the viewpoint position of the current frame, the lower the threshold for determining that there is no gaze operation is set. By setting the threshold in this way, vibrations near the recommended viewpoint can be suppressed.

[0046] <1-7-1. Processing procedure for application example> Fig. 9 is a flowchart showing the procedure of a display process according to an application example of the first embodiment. In the flowchart shown in Fig. 9, the same step numbers are assigned to steps in which the same processes as those in the flowchart shown in Fig. 7 are performed, while different step numbers are assigned to steps in which different processes are performed.

[0047] The display process shown in Fig. 9 differs from Fig. 7 in that processes from step S201 to step S203 are added. Specifically, step S201 can be executed after the process of step S103, just as an example.

[0048] For example, in step S201, the composite operation amount calculation unit 15C determines whether the operation movement amount calculated in step S102 exceeds a predetermined threshold, for example a threshold determined by the angle difference in the line of sight direction from the recommended viewpoint to the viewpoint position of the current frame.

[0049] Here, if the operation movement amount exceeds the threshold, i.e., if the operation movement amount is not equal to or less than the threshold (Yes in step S201), the composite operation amount calculation unit 15C calculates a composite movement amount by summing up the operation movement amount calculated in step S102 and the return movement amount calculated in step S103 for each of the three axes of roll and yaw (step S202).

[0050] On the other hand, if the operation movement amount does not exceed the threshold, that is, if the operation movement amount is equal to or less than the threshold (step S201 No), the composite operation amount calculation unit 15C considers that there is no gaze operation, that is, the operation movement amount of the three axes of roll and yaw is zero. That is, the composite operation amount calculation unit 15C calculates the return movement amount of the three axes of roll and yaw as the composite movement amount (step S203).

[0051] Thereafter, the process of the subsequent step S105 is executed based on the combined movement amount calculated in step S202 or step S203.

[0052] According to the application example of the first embodiment, vibrations in the vicinity of the recommended viewpoint can be suppressed.

[0053] <<2. Second embodiment>> In the above first embodiment, an example was given in which the sense of distance from the recommended viewpoint was conveyed by adjusting the amount of return movement, but the sense of distance from the recommended viewpoint can be conveyed without necessarily performing guidance control to return the viewpoint position to the recommended viewpoint. Therefore, as a second embodiment, an example will be given in which the sense of distance from the recommended viewpoint is conveyed by limiting the amount of operation movement.

[0054] <2-1. Example of functional configuration of display device> Fig. 10 is a block diagram showing an example of a functional configuration of a display device according to the second embodiment. As shown in Fig. 10, the display device 20 differs from the display device 10 shown in Fig. 3 in that it includes a transition control unit 21 that has a part of the processing contents different from those of the transition control unit 15.

[0055] <2-2. Transition control section> 3 in that the transition control unit 21 does not necessarily have functional units such as a return movement amount calculation unit 15B and a combined movement amount calculation unit 15C that correspond to the control of returning the viewpoint position to the recommended viewpoint. In addition, the transition control unit 21 differs from the transition control unit 15 shown in FIG. 3 in that the transition control unit 21 has a correction unit 21A that performs correction to limit the above-mentioned operation movement amount.

[0056] <2-2-1. Correction section> The correction unit 21A is a processing unit that corrects the above-mentioned operation movement amount based on the angle difference between the viewpoint position of the current frame and the line-of-sight direction of the recommended viewpoint.

[0057] 11 and 12 are diagrams showing an example of a method for correcting the operation movement amount. The horizontal axis of the graphs shown in FIG. 11 and FIG. 12 indicates the angle from the recommended viewpoint, and the horizontal axis indicates the operation movement amount. Furthermore, in FIG. 11 and FIG. 12, the operation movement amount before correction is shown by a dashed line, while the operation movement amount after correction is shown by a solid line. As shown in FIG. 11, a function that decreases the rate of monotonically increasing the operation movement amount after correction as the angle from the recommended viewpoint increases compared to the rate of monotonically increasing the operation movement amount before correction can be used for correcting the operation movement amount. As another example, as shown in FIG. 12, a function that linearly increases the operation movement amount before correction and the operation movement amount after correction with the same slope until the angle from the recommended viewpoint increases to a predetermined angle, and the slope of linearly increasing the operation movement amount after correction becomes smaller than the slope of linearly increasing the operation movement amount before correction when the angle from the recommended viewpoint becomes equal to or larger than the predetermined angle can be used for correcting the operation movement amount.

[0058] In this way, a function expressed in Log or a function expressed by a threshold value is used as a function that limits the increase in the amount of operation movement as the angle from the recommended viewpoint increases. This makes it possible to communicate the state of being far from the recommended viewpoint, that is, being out of range of the recommended viewpoint, by restricting the increase in the amount of operation movement as a limiter. Therefore, it is possible to visually feed back the recommended viewpoint without displaying various signs such as a frame or arrow indicating the location of the recommended viewpoint.

[0059] 11 and 12 show examples of functions that determine the operation movement amounts before and after correction, but the operation movement amounts do not necessarily have to be corrected according to a function. For example, it goes without saying that a lookup table or the like in which the operation movement amounts before and after correction are associated with each class of the rotation angle of the line of sight direction from the recommended viewpoint to the viewpoint position of the current frame can be used instead of a function.

[0060] <2-3. Display device processing procedure> Fig. 13 is a flowchart showing the procedure of the display process according to the second embodiment. In the flowchart shown in Fig. 13, the same step numbers are assigned to steps in which the same processes as those in the flowchart shown in Fig. 7 are performed, while different step numbers are assigned to steps in which different processes are performed.

[0061] The display process shown in Fig. 13 differs from Fig. 7 in that the processes of steps S103 and S104 shown in Fig. 7 are omitted and a process of step S301 is added. Specifically, step S301 can be executed after the process of step S102, as an example only. For example, in step S301, correction unit 21A corrects the operation movement amount calculated in step S102 based on the angle difference between the viewpoint position of the current frame and the line-of-sight direction of the recommended viewpoint.

[0062] After that, the process of the subsequent step S105 is executed based on the operation movement amount corrected in step S301.

[0063] <2-4. One aspect of the effect> As described above, the display device 20 of the present disclosure corrects the above-mentioned operation movement amount based on the angle difference between the viewpoint position of the current frame and the line of sight direction of the recommended viewpoint. Therefore, the display device 20 of the present disclosure allows the use of both the recommended viewpoint and the free viewpoint without displaying the position of the recommended viewpoint with a frame or the like.

[0064] <<3. Third embodiment>> In the above-mentioned second embodiment, an example was given in which the above-mentioned correction of the operation movement amount is incorporated while omitting the guidance control for returning the viewpoint position from the display device 10 according to the above-mentioned first embodiment to the recommended viewpoint, but the above-mentioned first embodiment and the above-mentioned second embodiment can be combined as they are. Hereinafter, as a third embodiment, an example will be given of a case in which the above-mentioned first embodiment and the above-mentioned second embodiment are combined as they are.

[0065] <3-1. Example of functional configuration of display device> Fig. 14 is a block diagram showing an example of a functional configuration of a display device according to the third embodiment. As shown in Fig. 14, a display device 30 differs from the display device 10 shown in Fig. 3 in that it includes a transition control unit 31 that has a part of the processing contents different from those of the transition control unit 15.

[0066] <3-2. Transition control section> The transition control unit 31 differs from the transition control unit 15 shown in FIG. 3 in that the transition control unit 31 further has a correction unit 21A shown in FIG. 10 in addition to the operation movement amount calculation unit 15A, the return movement amount calculation unit 15B, the composite movement amount calculation unit 15C, and the viewpoint position calculation unit 15D shown in FIG. 3.

[0067] <3-3. Display device processing procedure> Fig. 15 is a flowchart showing the procedure of the display process according to the third embodiment. In the flowchart shown in Fig. 15, the same step numbers are assigned to steps in which the same processes as those in the flowcharts shown in Fig. 7 and Fig. 13 are performed, while different step numbers are assigned to steps in which different processes are performed.

[0068] This process is also started in response to a user's operation to start playing a video clip, as an example. When the process is started in this manner, the display device 10 displays the recommended viewpoint, that is, the range (recommended range) corresponding to the RVP, as a display range, as shown in Fig. 15 (step S101).

[0069] Thereafter, the processes from step S102 to step S104 described below are performed for each of the three axes, pitch, roll, and yaw.

[0070] That is, the operation movement amount calculation unit 15A calculates the above operation movement amount based on the line-of-sight operation information output by the gyro sensor 14 (step S102). Next, the correction unit 21A corrects the operation movement amount calculated in step S102 based on the angle difference between the viewpoint position of the current frame and the line-of-sight direction of the recommended viewpoint (step S301). In addition, the return movement amount calculation unit 15B calculates the above return movement amount based on the recommended viewpoint information output by the decoder 12 (step S103).

[0071] Then, the composite movement amount calculation unit 15C calculates the composite movement amount by summing up the operation movement amount corrected in step S301 and the return movement amount calculated in step S103 for each of the three axes of roll and yaw (step S104).

[0072] Next, viewpoint position calculation unit 15D calculates the viewpoint position of the next frame by adding the line of sight direction angle corresponding to the composite movement amount calculated for each of the three axes of roll and yaw in step S104 to the line of sight direction of the current frame (step S105).

[0073] Then, the visual field rendering unit 13 causes the display unit 16 to display the display image with the visual field corresponding to the viewpoint position of the next frame calculated in step S105 as the display range of the projection image (step S106).

[0074] Thereafter, the processes from the above step S102 to the above step S106 are repeated until the video playback or distribution is ended (step S107 No). When the video playback or distribution is ended (step S107 Yes), the process is ended.

[0075] <3-4. One aspect of the effect> As described above, the display device 30 of the present disclosure can provide the effects of both the display device 10 according to the above-mentioned embodiment 1 and the display device 20 according to the above-mentioned embodiment 2. For example, the display device 30 of the present disclosure can convey a sense of distance to a recommended viewpoint, a sense of being out of range of a recommended viewpoint, and the like.

[0076] <<4. Fourth Embodiment>> In the above-described first to third embodiments, the process in the case where there is one recommended viewpoint has been exemplified. However, there may be a plurality of recommended viewpoints. Hereinafter, as a fourth embodiment, a process of selecting one recommended viewpoint from a plurality of recommended viewpoints will be exemplified.

[0077] <4-1. Example of Functional Configuration of Display Device> FIG. 16 is a block diagram showing an example of the functional configuration of a display device according to the fourth embodiment. As shown in FIG. 16, the display device 40 is different in that it has a transition control unit 41 whose processing content is partly different from that of the transition control unit 15 of the display device 10 shown in FIG. 3.

[0078] <4-2. Transition Control Unit> The transition control unit 41 is different in that it further has a score calculation unit 41A and a recommended viewpoint selection unit 41B as compared with the transition control unit 15 shown in FIG. 3.

[0079] <4-2-1. Score Calculation Unit> The score calculation unit 41A is a processing unit that calculates the score of each recommended viewpoint for each recommended viewpoint.

[0080] As an example only, the score calculation unit 41A can calculate the score of the recommended viewpoint according to the distance between the viewpoint position of the current frame and the recommended viewpoint according to the following formula (1). In the following formula (1), "S" i " refers to the score of the i-th recommended viewpoint. Also, in the following formula (1), "d" i " refers to the distance between the i-th recommended viewpoint and the viewpoint position of the current frame. FIG. 17 is a diagram showing an example of the distance used for score calculation. In FIG. 17, three recommended viewpoints RVP1 to RVP3 and the viewpoint position of the current frame are shown on the projection image. In the example shown in FIG. 17, the magnitude relationship of the distances d1, d2, and d3 between the three recommended viewpoints RVP1 to RVP3 and the viewpoint position of the current frame is "d1 < d3 < d2". Therefore, the magnitude relationship of the scores of the recommended viewpoints RVP1 to RVP3 is "S1 > S3 > S2".

[0081] S i = 1 / d i ···(1)

[0082] As another example, the score calculation unit 41A can calculate the score of the recommended viewpoint by assigning a predetermined weight to the distance between the viewpoint position of the current frame and the recommended viewpoint according to the following formula (2). "S" in the following formula (2) i " refers to the score of the i-th recommended viewpoint. Also, "d" in the following formula (2) i " refers to the distance between the i-th recommended viewpoint and the viewpoint position of the current frame. Furthermore, "w" in the following formula (2) i " refers to the weight of the i-th recommended viewpoint. For example, the weight of the recommended viewpoint may be set by a user such as a distributor like a director or an end user who is a viewer. Also, by referring to the viewing history of the distribution data of multiple viewers, it is possible to automatically set a larger weight for recommended viewpoints with a larger number of viewers.

[0083] S i = w i / d i ···(2)

[0084] FIG. 18 is a diagram showing an example of the distance and weight used for score calculation. In FIG. 18, similar to FIG. 17, three recommended viewpoints RVP1 to RVP3 and the viewpoint position of the current frame are shown on the projection image. Furthermore, in FIG. 18, the weights of the recommended viewpoints RVP1 to RVP3 are shown by the size of the circles. In the example shown in FIG. 18, the magnitude relationship of the distances d1, d2, and d3 between the three recommended viewpoints RVP1 to RVP3 and the viewpoint position of the current frame is "d1 < d3 < d2". On the other hand, the magnitude relationship of the weights of the three recommended viewpoints RVP1 to RVP3 is "w1 = w2 < w3". In this case, the magnitude relationship of the scores of the recommended viewpoints RVP1 to RVP3 does not necessarily become "S1 > S3 > S2". That is, depending on the weight w3 of the recommended viewpoint RVP3, the magnitude relationship of the scores of the recommended viewpoints RVP1 to RVP3 may become "S3 > S1 > S2" as shown in FIG. 18.

[0085] <4-2-2. Recommended viewpoint selection section> The recommended viewpoint selection unit 41B is a processing unit that selects one recommended viewpoint from among a plurality of recommended viewpoints.

[0086] As an example, the recommended viewpoint selection unit 41B selects the recommended viewpoint with the maximum score from among the multiple recommended viewpoints. For example, in the example shown in Fig. 17, the recommended viewpoint RVP1 is selected from among the recommended viewpoints RVP1 to RVP3, while in the example shown in Fig. 18, the recommended viewpoint RVP3 is selected from among the recommended viewpoints RVP1 to RVP3.

[0087] By selecting the recommended viewpoint with the highest score in this manner, it is possible to recommend the nearest recommended viewpoint or the recommended viewpoint that is the most interesting in the distribution data.

[0088] In this example, the higher the score value, the higher the priority of the recommended viewpoint is calculated, but the present invention is not limited to this. For example, the lower the score value, the higher the priority of the recommended viewpoint is calculated. In this case, the recommended viewpoint with the smallest score may be selected.

[0089] <4-3. Display device processing procedure> Fig. 19 is a flowchart showing the procedure of the display process according to the fourth embodiment. In the flowchart shown in Fig. 19, the same step numbers are assigned to steps in which the same processes as those in the flowchart shown in Fig. 7 are performed, while different step numbers are assigned to steps in which different processes are performed.

[0090] The display process shown in Fig. 19 differs from Fig. 7 in that the processes of step S401 and step S402 are added following the process of step S101 shown in Fig. 7. Specifically, in step S401, as an example only, the score calculation unit 41A calculates the score of each recommended viewpoint. After that, the recommended viewpoint selection unit 41B selects one recommended viewpoint from among the multiple recommended viewpoints based on the score calculated in step S401 (step S402).

[0091] Thereafter, the process of step S102 is executed for the recommended viewpoint selected in step S402.

[0092] <4-4. One aspect of the effect> As described above, the display device 40 of the present disclosure calculates a score for each recommended viewpoint and selects one recommended viewpoint from among multiple recommended viewpoints based on the score. Therefore, according to the display device 40 of the present disclosure, even when multiple recommended viewpoints coexist, it is possible to narrow down to one recommended viewpoint.

[0093] <<5. Modifications>> Modifications of the first to fourth embodiments will be illustrated below.

[0094] <5-1. Non-visual feedback> In the first to fourth embodiments, examples have been given in which the position of the recommended viewpoint is transmitted by the transition of the display range of the projection image, but the position of the recommended viewpoint can also be transmitted by methods other than display. For example, the position of the recommended viewpoint can be transmitted through audio output. As an example, the volume can be lowered the farther away from the recommended viewpoint, or a sound image can be localized at the recommended viewpoint using stereophonic sound to make the user perceive a sound source in the direction of the recommended viewpoint. In addition to such audio output, when the distance of the viewpoint position of the current frame or the next frame is greater than a predetermined value from the recommended viewpoint, a vibrator or the like can be activated to make the user perceive vibration, or a gyroscope can be used to make the user perceive a larger weight as the user moves away from the recommended viewpoint.

[0095] <5-2. Conditions for Execution of Guidance Control> In the first, third and fourth embodiments, the guidance control for returning the viewpoint position to the recommended viewpoint is performed every frame, but a certain condition can be set. For example, the display device 10, 30 or 40 may execute the guidance control for returning the viewpoint position to the recommended viewpoint only when the operation movement amount is zero or a predetermined threshold or less, that is, when there is no gaze operation. For example, when the operation movement amount is zero or a predetermined threshold or less, the return movement amount calculation unit can be caused to calculate the return movement amount.

[0096] <5-3. Function implementation subject> In the first to fourth embodiments, examples have been given in which the transition control units 15, 21, 31, and 41 are mounted on the display devices 10, 20, 30, and 40, but the transition control units 15, 21, 31, and 41 may be mounted on the distribution server 9. In this case, the gaze operation information by the gyro sensor 14 may be acquired from the display device 10, 20, 30, or 40. In this way, in addition to the display devices 10, 20, 30, and 40, the distribution server 9 may also correspond to an example of an image processing device.

[0097] <5-4. Other variations> Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0098] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0099] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be provided.

[0100] <<6. Hardware Configuration>> The series of processes in the above-mentioned display device 10, 20, 30, or 40 can be executed by hardware or software. When the series of processes are executed by software, a program constituting the software is installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0101] FIG. 20 is a block diagram showing an example of the hardware configuration of the computer 400. As shown in FIG.

[0102] In the computer 400, a central processing unit (CPU) 401, a read only memory (ROM) 402, and a random access memory (RAM) 403 are interconnected via a bus 404.

[0103] An input / output interface 405 is further connected to the bus 404. An input unit 406, an output unit 407, a recording unit 408, a communication unit 409, and a drive 410 are connected to the input / output interface 405.

[0104] The input unit 406 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 407 includes a display, a speaker, etc. The recording unit 408 includes a hard disk, a non-volatile memory, etc. The communication unit 409 includes a network interface, etc. The drive 410 drives a removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0105] In the computer 400 configured as above, the CPU 401 loads, for example, a program recorded in the recording unit 408 into the RAM 403 via the input / output interface 405 and the bus 404, and executes the program, thereby performing the series of processes described above.

[0106] The program executed by the computer 400 (CPU 401) can be provided by being recorded on a removable medium 411 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0107] In the computer, the program can be installed in the recording unit 408 via the input / output interface 405 by mounting the removable medium 411 in the drive 410. The program can also be received by the communication unit 409 via a wired or wireless transmission medium and installed in the recording unit 408. Alternatively, the program can be pre-installed in the ROM 402 or the recording unit 408.

[0108] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0109] The present technology can also be configured as follows. (1) an output unit that outputs a part of an image including the recommended viewpoint information to a display unit as a display image; a transition control unit that transitions a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, The output unit outputs a part of the image to the display unit based on the transitioned display range. Image processing device. (2) The transition control unit an operation movement amount calculation unit that calculates an operation movement amount for moving a viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; a return movement amount calculation unit that calculates a return movement amount for returning a viewpoint position corresponding to the display image to a viewpoint position corresponding to the recommended viewpoint information; a viewpoint position calculation unit that calculates a viewpoint position of a next frame to be outputted next to the frame of the display image outputted to the display unit based on the operation movement amount and the return movement amount, The output unit is outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame; The image processing device according to (1) above. (3) the return movement amount calculation unit calculates the return movement amount based on an angle difference of a line of sight from a viewpoint position corresponding to the recommended viewpoint information to a viewpoint position corresponding to the display image; The image processing device according to (2) above. (4) the return movement amount calculation unit sets a speed at which the camera returns to the viewpoint position corresponding to the recommended viewpoint information to be increased as the angular difference in the line of sight direction increases. The image processing device according to (3) above. (5) the return movement amount calculation unit linearly increases a speed of returning to the viewpoint position corresponding to the recommended viewpoint information as the angular difference in the line of sight direction increases; The image processing device according to (4) above. (6) the return movement amount calculation unit monotonically increases a speed of returning to the viewpoint position corresponding to the recommended viewpoint information and decreases a rate of the monotonous increase as the angular difference in the line of sight direction increases. The image processing device according to (4) above. (7) the viewpoint position calculation unit calculates a viewpoint position of the next frame based on the return movement amount when the operation movement amount is equal to or less than a predetermined threshold, and calculates a viewpoint position of the next frame based on the operation movement amount and the return movement amount when the operation movement amount is not equal to or less than the threshold. The image processing device according to any one of (2) to (6). (8) The threshold value is set to a higher value as the angle difference of the line of sight from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image is smaller. The image processing device according to (7) above. (9) The transition control unit an operation movement amount calculation unit that calculates an operation movement amount for moving a viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; a correction unit that corrects the operation movement amount based on an angle difference in a line-of-sight direction from a viewpoint position corresponding to the recommended viewpoint information to a viewpoint position corresponding to the display image; a viewpoint position calculation unit that calculates a viewpoint position of a next frame to be outputted next to the frame of the display image outputted to the display unit based on the corrected operation movement amount, The output unit is outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame; The image processing device according to any one of (1) to (8). (10) the correction unit reduces a rate of monotonous increase in the amount of operation movement after the correction compared to a rate of monotonous increase in the amount of operation movement before the correction as the angular difference in the line of sight direction increases. The image processing device according to (9) above. (11) the correction unit linearly increases the pre-correction operation movement amount and the corrected operation movement amount with the same slope until the angular difference in the line-of-sight directions increases to a predetermined angle, and when the angular difference in the line-of-sight directions becomes equal to or greater than the predetermined angle, makes the slope at which the corrected operation movement amount increases linearly smaller than the slope at which the pre-correction operation movement amount increases linearly. The image processing device according to (9) above. (12) a score calculation unit that calculates a score for each viewpoint position included in the recommended viewpoint information; a recommended viewpoint selection unit that selects one of the viewpoint positions included in the recommended viewpoint information based on the score, the return movement amount calculation unit calculates a return movement amount for returning a viewpoint position corresponding to the display image to a viewpoint position selected from among viewpoint positions included in the recommended viewpoint information; The image processing device according to any one of (2) to (11). (13) the score calculation unit calculates the score based on a distance between a viewpoint position included in the recommended viewpoint information and a viewpoint position corresponding to the display image. The image processing device according to (12) above. (14) The score calculation unit decreases the value of the score as the distance increases. The image processing device according to (13) above. (15) The score calculation unit calculates the score by applying a predetermined weight to the distance. The image processing device according to (13) above. (16) The score calculation unit increases the value of the score as the weight increases, or decreases the value of the score as the weight decreases. The image processing device according to (15) above. (17) The weight is assigned a larger value to a recommended viewpoint having a larger number of viewers. The image processing device according to (16) above. (18) The return movement amount calculation unit executes calculation of the return movement amount when the operation movement amount is equal to or less than a predetermined threshold value. The image processing device according to any one of (2) to (17). (19) A part of the image including the recommended viewpoint information is output to a display unit as a display image; transitioning a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information; The computer executes the process, The process of outputting the image to the display unit includes outputting a part of the image to the display unit based on the transitioned display range. Image processing methods. (20) A part of the image including the recommended viewpoint information is output to a display unit as a display image; transitioning a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information; The process is executed by a computer, The process of outputting the image to the display unit includes outputting a part of the image to the display unit based on the transitioned display range. program. [Explanation of symbols]

[0110] 1. Image distribution system 3. Multi-camera 5 Wide viewing angle image conversion section 7 Encoders 9. Distribution Server 10 Display device 11 Receiving section 12 Decoder 13 Visual Field Drawing Section 14 Gyro sensor 15 Transition control section 15A Operation movement amount calculation section 15B Return movement amount calculation section 15C Combined movement amount calculation section 15D Viewpoint position calculation unit 16 Display section

Claims

1. An output unit that outputs a part of an image including recommended viewpoint information to a display unit as a display image; a transition control unit that transitions a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information, The output unit outputs a portion of the image to the display unit based on the transitioned display range, The transition control unit an operation movement amount calculation unit that calculates an operation movement amount for moving a viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; a return movement amount calculation unit that calculates a return movement amount for returning a viewpoint position corresponding to the display image to a viewpoint position corresponding to the recommended viewpoint information; a viewpoint position calculation unit that calculates a viewpoint position of a next frame to be outputted next to the frame of the display image outputted to the display unit based on the operation movement amount and the return movement amount, The output unit is outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame; Image processing device.

2. the return movement amount calculation unit calculates the return movement amount based on an angle difference of a line of sight from a viewpoint position corresponding to the recommended viewpoint information to a viewpoint position corresponding to the display image; The image processing device according to claim 1 .

3. the return movement amount calculation unit sets a speed at which the camera returns to the viewpoint position corresponding to the recommended viewpoint information to be increased as the angular difference in the line of sight direction increases. The image processing device according to claim 2 .

4. the return movement amount calculation unit linearly increases a speed of returning to the viewpoint position corresponding to the recommended viewpoint information as the angular difference in the line of sight direction increases; The image processing device according to claim 3 .

5. the return movement amount calculation unit monotonically increases a speed of returning to the viewpoint position corresponding to the recommended viewpoint information and decreases a rate of the monotonous increase as the angular difference in the line of sight direction increases. The image processing device according to claim 3 .

6. the viewpoint position calculation unit calculates a viewpoint position of the next frame based on the return movement amount when the operation movement amount is equal to or less than a predetermined threshold, and calculates a viewpoint position of the next frame based on the operation movement amount and the return movement amount when the operation movement amount is not equal to or less than the threshold. The image processing device according to claim 1 .

7. The threshold value is set to a higher value as the angle difference of the line of sight from the viewpoint position corresponding to the recommended viewpoint information to the viewpoint position corresponding to the display image is smaller. The image processing device according to claim 6.

8. a score calculation unit that calculates a score for each viewpoint position included in the recommended viewpoint information; a recommended viewpoint selection unit that selects one of the viewpoint positions included in the recommended viewpoint information based on the score, the return movement amount calculation unit calculates a return movement amount for returning a viewpoint position corresponding to the display image to a viewpoint position selected from among viewpoint positions included in the recommended viewpoint information; The image processing device according to claim 1 .

9. the score calculation unit calculates the score based on a distance between a viewpoint position included in the recommended viewpoint information and a viewpoint position corresponding to the display image. The image processing device according to claim 8.

10. The score calculation unit decreases the value of the score as the distance increases. The image processing device according to claim 9 .

11. The score calculation unit calculates the score by applying a predetermined weight to the distance. The image processing device according to claim 9 .

12. The score calculation unit increases the value of the score as the weight increases, or decreases the value of the score as the weight decreases. The image processing device according to claim 11.

13. The weight is assigned a larger value to a recommended viewpoint having a larger number of viewers. The image processing device according to claim 12.

14. The return movement amount calculation unit executes calculation of the return movement amount when the operation movement amount is equal to or less than a predetermined threshold value. The image processing device according to claim 1 .

15. A part of the image including the recommended viewpoint information is output to a display unit as a display image; transitioning a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information; The computer executes the process, The process of outputting the image to the display unit includes outputting a part of the image to the display unit based on the transitioned display range, The transition process includes: calculating an operation movement amount for moving a viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; A return movement amount for returning a viewpoint position corresponding to the display image to a viewpoint position corresponding to the recommended viewpoint information is calculated; calculating a viewpoint position of a next frame to be outputted next to the frame of the display image outputted to the display unit based on the operation movement amount and the return movement amount; The process of outputting to the display unit includes: outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame; Image processing methods.

16. A part of the image including the recommended viewpoint information is output to a display unit as a display image; transitioning a display range of the image based on a positional relationship between a viewpoint position corresponding to the display image output to the display unit and a viewpoint position corresponding to the recommended viewpoint information; The process is executed by a computer, The process of outputting the image to the display unit includes outputting a part of the image to the display unit based on the transitioned display range, The transition process includes: calculating an operation movement amount for moving a viewpoint position corresponding to the display image in accordance with a viewpoint operation based on viewpoint operation information; A return movement amount for returning a viewpoint position corresponding to the display image to a viewpoint position corresponding to the recommended viewpoint information is calculated; calculating a viewpoint position of a next frame to be outputted next to the frame of the display image outputted to the display unit based on the operation movement amount and the return movement amount; The process of outputting to the display unit includes: outputting a part of the image to the display unit based on the display range determined from the viewpoint position of the next frame; program.

Images