Video display processing device and program

Abstract

To complementarily present a 3DCG model video and a 360-degree video and geometrically match both videos so that a portion of the 3DCG model video is not missing, thus realizing visual continuity.SOLUTION: A state management unit 12 of a transition function unit 2 of a video display processing device 1 causes a transition state SS to go to an expansion-in-progress state S2 when a distance (t) is smaller than a transition start threshold dist_th_in, and generates a control signal that indicates the initiation of a 360-degree video, forefront material, expansion, and hiding of other objects. On an attached spherical object 3 are set the size of a screen sphere that gradually increases, a spherical video that indicates a playback 360-degree video, and a forefront material, etc., with display allowability that indicates hiding and similar other information being set on other objects. Thus, a transition from the world of the 3DCG model video to the world of the 360-degree video occurs.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image display processing device and a program for displaying images and 360-degree images of 3DCG (three-dimensional computer graphics) models created by photogrammetry. [Background technology]

[0002] Recently, there has been an accelerating trend to use photogrammetry to create 3DCG models of disaster sites and the like (see, for example, Non-Patent Document 1). Photogrammetry is a method of generating a three-dimensional 3DCG model on a computer using sequential still images extracted from a large number of photographs, videos, etc.

[0003] In photogrammetry, feature points of a subject are extracted from a photograph or the like, and the three-dimensional positions of the feature points and the photographing location are identified by triangulation of the same feature points extracted from multiple photographs or the like.

[0004] For this reason, photogrammetry has the drawback that when the positions of feature points of a subject in multiple photographs move or their appearance changes, parts of the generated 3DCG model may be missing or deformed. Subjects whose feature points move include flags and animals, while subjects whose appearance changes include reflective objects such as mirrors and transparent objects such as water.

[0005] Meanwhile, particularly in the field of virtual reality (VR), the use of 360-degree cameras to capture spherical images (360-degree images) in all directions from a single point is becoming popular. Cameras capable of capturing high-definition 360-degree images with a horizontal resolution of 11K (monocular) or 10K (binocular, stereoscopic) are also available (see, for example, Non-Patent Document 2).

[0006] These cameras can be used not only to shoot 360-degree images, but also to generate 3DCG models using photogrammetry. Images taken with such cameras are stitched together, combining multiple images taken with multiple lenses into a single image. However, this image is essentially a representation of the outside world as it is, and looks more realistic than a 3DCG model created using photogrammetry.

[0007] Unlike 3DCG model images created using photogrammetry, 360-degree video does not allow users to move their viewpoint up, down, left, right, forward, or backward while viewing. As a result, users viewing 360-degree video have a viewing experience that is fixed to the shooting position (a 3DoF (three degrees of freedom of rotation) experience, which will be described later). [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] "Atami Debris Flow Disaster: Volunteers Release 3D Model of Site Created from Drone Images," [online], July 5, 2021, ITmedia, Inc., [Retrieved October 7, 2021], Internet<https: / / www.itmedia.co.jp / news / articles / 2107 / 05 / news154.html> [Non-patent document 2] “Insta360 TITAN”, [online], [searched October 7, 2021], Internet<https: / / www.insta360.com / jp / product / insta360-titan> Summary of the Invention [Problem to be solved by the invention]

[0009] As mentioned above, images of 3DCG models created using photogrammetry have the disadvantage that parts of the model may be missing or distorted.

[0010] For example, in reporting on a disaster, if a part of the image of a 3DCG model created using photogrammetry is missing, it is difficult to determine whether this is due to a flaw in photogrammetry or whether the situation is actually caused by the disaster.

[0011] In order to solve this problem, the inventors came up with the idea that by presenting an image of a 3DCG model created by photogrammetry and the aforementioned 360-degree image, it would be possible to compensate for the drawback of parts of the 3DCG model image being missing, etc.

[0012] Users who view images of 3DCG models created using photogrammetry can experience 6DoF (six degrees of freedom: three degrees of freedom for rotation and three degrees of freedom for translation), while users who view 360-degree images can experience 3DoF.

[0013] However, in a virtual space where an image of a 3DCG model created using photogrammetry is placed, if the 360-degree image is simply placed at the shooting position of the 360-degree image, the two images will not match geometrically, resulting in a lack of visual continuity.

[0014] Therefore, the present invention has been made to solve the above-mentioned problems, and its purpose is to provide an image display processing device and program that can compensate for the drawback of parts of the 3DCG model image being missing, etc., by presenting a photogrammetric 3DCG model image and a 360-degree image in a complementary manner, geometrically matching the two images, and achieving visual continuity. [Means for solving the problem]

[0015] In order to solve the above problem, the video display processing device of claim 1 arranges a plurality of spherical objects in a virtual space in which a plurality of 3DCG objects are arranged by photogrammetry, and displays a 3DCG model video including the 3DCG object that is within the field of view of the user among the plurality of 3DCG objects according to the viewpoint position and line of sight of the user in the virtual space, and displays a 360-degree video of the spherical object that is within the field of view of the user among the plurality of spherical objects as an icon, and is equipped with a plurality of transition function units that are attached to one corresponding spherical object among the plurality of spherical objects, and that sequentially transition the transition state of the corresponding spherical object from waiting S1, expanding S2, expanded S3, shrinking S4, and reduced S5, and set predetermined data according to the state of the transition to the spherical object, and is attached to one corresponding transition function unit among the plurality of transition function units, and the predetermined data is set by the corresponding transition function unit. a camera object in which predetermined data including the user's viewpoint position and line of sight is set; and a renderer that identifies the spherical object and the 3DCG object that are within the user's field of view based on the user's viewpoint position and line of sight set in the camera object, renders the identified spherical object, and also renders the 3DCG model object including the identified 3DCG object, generates a display image, and outputs it to a display device; and when the user is away from the corresponding spherical object by a first predetermined distance or more, the transition function unit sets the state of the transition to standby S1, and sets data to the spherical object for displaying a still image of the 360-degree image as a thumbnail, and when the user gets closer to the spherical object than the first predetermined distance,The state of the transition is changed from the waiting S1 to the expanding S2, and data for gradually expanding the 360-degree video and displaying it in the foreground is set to the spherical object. When the size of the spherical object reaches a first predetermined size, the state of the transition is changed from the expanding S2 to the expanded S3, and data for stopping the expansion of the 360-degree video is set to the spherical object. When there is a key input from the user or when playback of the 360-degree video is completed, the state of the transition is changed from the expanded S3 to the shrinking S4. and sets data for gradually reducing and displaying the 360-degree video in the spherical object, and when the size of the spherical object becomes smaller than a second predetermined size, the state of the transition is transitioned from the reducing S4 to the reduced S5, and data for stopping the reduction of the 360-degree video and canceling the application of the foreground display is set in the spherical object, and when the user moves away from the corresponding spherical object by more than a second predetermined distance, the state of the transition is transitioned from the reduced S5 to the waiting S1.

[0016] and a 3DCG model object including a plurality of 3DCG objects, and a position of the 3DCG model object and a display / non-display information indicating whether to display or not the image of the spherical object are set for each of the plurality of 3DCG objects. The 3DCG model object is an object including the plurality of 3DCG objects, and the position of the 3DCG object and the display / non-display information indicating whether to display or not the image of the spherical object are set for each of the plurality of 3DCG objects. The transition function unit sets the state of the transition to standby S1 when the user is away from the corresponding spherical object by a first predetermined distance or more, and a start frame of the 360-degree image is displayed. the spherical object, and when the user approaches the spherical object closer than the first predetermined distance, the state of the transition is changed from the waiting state S1 to the expanding state S2, and the size for gradually expanding the image of the spherical object, the spherical image showing the played 360-degree image when the played 360-degree image is set as the played 360-degree image, and a foreground material indicating that the played 360-degree image will be displayed in the foreground are set to the spherical object, and display / non-display information indicating that the image is not displayed is set to the spherical object. , setting the size of the spherical object other than the spherical object among the plurality of spherical objects and the plurality of 3DCG objects, when the size of the spherical object reaches a first predetermined size, transitioning the state of the transition from the Expanding S2 to the Enlarged S3, and setting the size for stopping the expansion of the image of the spherical object to the spherical object, when there is a key input from the user or when playback of the 360-degree video is completed, transitioning the state of the transition from the Enlarged S3 to the Shrinking S4,the size for gradually reducing the image of the spherical object is set to the spherical object, and displayability information indicating display is set to the spherical objects other than the spherical object among the plurality of spherical objects and to the plurality of 3DCG objects; when the size of the spherical object becomes smaller than a second predetermined size, the state of the transition is transitioned from the reducing S4 to the reduced S5, and the size for stopping the reduction of the image of the spherical object, the spherical image showing a still image of the first frame of the 360-degree image, and an unlit material with the foreground material no longer applied are set to the spherical object; when the user moves away from the corresponding spherical object more than a second predetermined distance, the state of the transition is transitioned from the reduced S5 to the waiting S1; and when the state of the transition is the waiting S1, the renderer pastes the still image inside the spherical surface of the spherical object, and performs culling to exclude the outside of the sphere and only target the inside of the sphere, and performs culling to select the viewpoint position and the a thumbnail of the icon sphere as seen from the user's viewpoint is generated by rendering in the viewing direction, and the display video including the thumbnail of the icon sphere is generated; when the state of the transition is expanding S2, the playback 360-degree video is attached to the inside of the sphere for the spherical object, and rendering is performed at the viewpoint position and viewing direction set for the camera object to generate a video of a screen sphere that grows over time according to the size and is displayed in the foreground in accordance with the foreground material; when the state of the transition is expanded S3, the spherical object is rendered to stop the expansion and generate a video of a screen sphere that is displayed in the foreground;the display video is generated, which does not include images of the spherical objects other than the spherical object and images of the plurality of 3DCG objects, and in which the image of the screen sphere is displayed in the foreground; when the state of the transition is in the reduced state S4, for the spherical object, an opening frame of the played 360-degree video is attached to the inside of the sphere, and rendering is performed at the viewpoint position and the line of sight direction set for the camera object, thereby generating an image of the screen sphere that becomes smaller over time according to the size and is displayed in the foreground; for the specified spherical objects other than the spherical object and the plurality of 3DCG objects, rendering is performed in accordance with display availability information indicating the display, and the display video is generated, which includes images of the spherical objects other than the specified spherical object and images of the plurality of 3DCG objects, and in which the image of the screen sphere is displayed in the foreground; when the state of the transition is in the reduced state S5, rendering is performed for the spherical object, thereby generating a thumbnail of the icon sphere, and generating the display video including a thumbnail of the icon sphere with the foreground display application canceled.

[0017] Further, a video display processing device according to claim 3 is the video display processing device according to claim 2, wherein the transition function unit includes a distance calculation unit that calculates a distance distance(t) between the user's viewpoint position set in the camera object and the spherical object position set in the spherical object, using the user's viewpoint position set in the camera object and the spherical object position set in the spherical object; a key input unit that outputs a key control signal in response to a key operation by the user; a scaling unit that calculates a scaling factor scale_magni(t) as the size of the image of the spherical object by dividing the radius of the screen sphere by a preset radius of the icon sphere; and when the distance distance(t) calculated by the distance calculation unit is smaller than a preset transition start threshold dist_th_in, transitions the state of the transition from the waiting S1 to the expanding S2; a state management unit that transitions the state of the transition from Enlarging S2 to Enlarged S3 when the enlargement / reduction rate scale_magni(t) calculated by the enlargement / reduction unit is equal to or greater than a preset enlargement completion threshold max_scale_magni, transitions the state of the transition from Enlarged S3 to Reduced S4 when the key control signal output from the key input unit is input or when playback of the 360-degree video is completed, transitions the state of the transition from Reduced S4 to Reduced S5 when the enlargement / reduction rate scale_magni(t) is smaller than a preset reduction completion threshold min_scale_magni, and transitions the state of the transition from Reduced S5 to Waiting S1 when the distance distance(t) is greater than a preset transition completion threshold dist_th_out.

[0018] A video display processing device according to claim 4 is the video display processing device according to claim 3, characterized in that the transition completion threshold dist_th_out is set to a value greater than the transition start threshold dist_th_in.

[0019] A video display processing device according to claim 5 is the video display processing device according to claim 3 or 4, characterized in that the reduction completion threshold min_scale_magni is set to 1.0.

[0020] Furthermore, a video display processing device according to claim 6 is the video display processing device according to any one of claims 3 to 5, wherein the transition function unit further comprises a material change unit, a video playback processing unit, and an other object display control unit, and when the state management unit transitions the state of the transition from the standby S1 to the expanding S2, the state management unit sends a control signal indicating the start of the 360-degree video to the video playback processing unit, a control signal indicating the foreground material to the material change unit, a control signal indicating expansion to the expansion / reduction unit, and a control signal indicating the non-display to the other object. When the state of the transition is transitioned from the enlarged S2 to the enlarged S3, a control signal indicating enlargement stop is output to the enlargement / reduction unit, when the state of the transition is transitioned from the enlarged S3 to the reduced S4, a control signal indicating cueing of the 360-degree video is output to the video playback processing unit, a control signal indicating reduction is output to the enlargement / reduction unit, and a control signal indicating the display is output to the other object display control unit, when the state of the transition is transitioned from the reduced S4 to the reduced S5, a control signal indicating reduction stop is output to the video playback processing unit, a control signal indicating reduction is output to the enlargement / reduction unit, and a control signal indicating the display is output to the other object display control unit, a control signal indicating the end of the 360-degree video to the video playback processing unit, and a control signal indicating the unlit material to the material change unit; when the scaling unit receives a control signal indicating the expansion from the state management unit, it calculates the size so that the video of the spherical object gradually expands and sets the size to the spherical object; when it receives a control signal indicating the stop of the expansion, it stops the calculation of the size so that the expansion of the video of the spherical object stops and sets the size to the spherical object; when it receives a control signal indicating the contraction, it calculates the size so that the video of the spherical object gradually contracts and sets the size to the spherical object; when it receives a control signal indicating the contraction stop, it stops the calculation of the size so that the contraction of the video of the spherical object stops and sets the size to the spherical object; when it receives a control signal indicating the foreground material from the state management unit, the material change unit sets the foreground material to the spherical object; and when it receives a control signal indicating the unlit material,The unlit material is set to the spherical object, and when the video playback processing unit receives a control signal from the state management unit indicating the start of the 360-degree video, it plays the 360-degree video and sets the spherical video representing the played 360-degree video to the spherical object, and when it receives a control signal indicating a cue to the 360-degree video, it cue the 360-degree video and sets the spherical video representing the opening frame of the played 360-degree video to the spherical object, and when it receives a control signal indicating the end of the 360-degree video, it extracts the opening frame from the 360-degree video and sets the spherical video representing the opening frame ... the spherical image indicating a still image is set to the spherical object, and when the other object display control unit receives a control signal indicating the non-display from the state management unit, the other object display control unit sets displayability information indicating the non-display to the spherical objects other than the spherical object and the plurality of 3DCG objects among the plurality of spherical objects, and when the other object display control unit receives a control signal indicating the display from the state management unit, the other object displayability information indicating the display to the spherical objects other than the spherical object and the plurality of 3DCG objects among the plurality of spherical objects.

[0021] Furthermore, the video display processing device of claim 7 is the video display processing device of claim 6, wherein the video playback processing unit is configured such that one of a loop setting, a stop setting, and an end setting is preset as the operating mode, and when the loop setting is preset, once playback of the 360-degree video is completed, playback continues from the first frame of the 360-degree video, when the stop setting is preset, once playback of the 360-degree video is completed, playback pauses at the final frame of the 360-degree video, when the end setting is preset, once playback of the 360-degree video is completed, a playback state of playback completion is output to the state management unit, and when the state management unit receives the playback state of playback completion from the video playback processing unit, it sets the state of the transition to S4.

[0022] Furthermore, the video display processing device of claim 8 is characterized in that, in the video display processing device of claim 2, the transition function unit further sets audio information indicating a sound effect corresponding to the state of the transition or the transition of the state to the spherical object, and the renderer plays the sound effect of the audio information set to the spherical object.

[0023] Furthermore, the video display processing device of claim 9 is characterized in that, in the video display processing device of claim 2 or 8, the transition function unit further sets motion information indicating an action to the spherical object when the state of the transition is the standby S1, and the renderer moves the display of the icon ball up and down at a predetermined speed when the motion information set to the spherical object indicates the action.

[0024] Furthermore, the program of claim 10 provides a computer constituting an image display processing device that arranges a plurality of spherical objects in a virtual space in which a plurality of 3DCG objects are arranged by photogrammetry, and displays a 3DCG model image including the 3DCG object among the plurality of 3DCG objects that is within the field of view of the user according to the viewpoint position and line of sight of the user in the virtual space, and also displays a 360-degree image of the spherical object among the plurality of spherical objects that is within the field of view of the user as an icon, the computer being provided with a plurality of transition function units each having one corresponding spherical object attached thereto, sequentially transitioning the transition state of the corresponding spherical object from waiting S1, expanding S2, expanded S3, shrinking S4, and reduced S5, and setting predetermined data according to the state of the transition to the spherical object, the computer being attached to one corresponding transition function unit among the plurality of transition function units, and the predetermined data being set by the corresponding transition function unit. a 3DCG model object including the plurality of 3DCG objects, with predetermined data set for each of the plurality of 3DCG objects; a camera object for which predetermined data including the user's viewpoint position and line of sight is set; and a program for causing the program to function as a renderer that identifies the spherical object and the 3DCG object that are within the field of view of the user, based on the user's viewpoint position and line of sight set in the camera object, renders the identified spherical object, and also renders the 3DCG model object including the identified 3DCG object, generates a display image, and outputs it to a display device, wherein the program causes the program to function as a renderer that, when the user is away from the corresponding spherical object by a first predetermined distance or more, sets data to the spherical object for displaying a still image of the 360-degree image as a thumbnail, and when the user is away from the spherical object by a first predetermined distance or more, sets data to the spherical object for displaying a still image of the 360-degree image as a thumbnail, andWhen the user approaches closer than the first predetermined distance, the state of the transition is changed from the waiting state S1 to the expanding state S2, and data for gradually expanding the 360-degree video and displaying it in the foreground is set to the spherical object. When the size of the spherical object reaches a first predetermined size, the state of the transition is changed from the expanding state S2 to the expanded state S3, and data for stopping the expansion of the 360-degree video is set to the spherical object. When the user inputs a key or when playback of the 360-degree video is completed, the state of the transition is changed to the expanded state S3. the state of the transition is transitioned from Scaling S4 to the Reduced S5, data for gradually reducing and displaying the 360-degree video is set to the spherical object when the size of the spherical object becomes smaller than a second predetermined size, data for stopping the reduction of the 360-degree video and canceling the application of the foreground display is set to the spherical object when the size of the spherical object becomes smaller than a second predetermined size, the state of the transition is transitioned from Scaling S4 to the Reduced S5, and data for stopping the reduction of the 360-degree video and canceling the application of the foreground display is set to the spherical object when the user moves away from the corresponding spherical object by more than a second predetermined distance, the state of the transition is transitioned from Reduced S5 to the Waiting S1. [Effects of the Invention]

[0025] As described above, according to the present invention, by presenting a photogrammetric 3DCG model image and a 360-degree image in a complementary manner, it is possible to compensate for the drawback of parts of the 3DCG model image being missing, etc. Furthermore, it is possible to geometrically match both images and achieve visual continuity. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a block diagram showing an example of the configuration of a video display processing device according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an example of the data structure of a spherical object. [Figure 3] FIG. 2 is a block diagram showing a configuration example of a transition function unit. [Figure 4] FIG. 10 is a diagram illustrating state transitions of a spherical object. [Figure 5] 10 is a flowchart illustrating an example of processing by a state management unit. [Figure 6] 10 is a flowchart showing an example of processing during enlargement (step S505). [Figure 7] 10 is a flowchart showing an example of enlargement processing (step S509). [Figure 8] 10 is a flowchart showing an example of processing during reduction (step S513). [Figure 9] 10 is a flowchart showing an example of a reduction process (step S517). [Figure 10] FIG. 10 is a block diagram showing a first modified example of the configuration example of the transition function unit. [Figure 11] FIG. 10 is a block diagram showing a second modified example of the configuration example of the transition function unit. [Figure 12] 10A and 10B are diagrams illustrating states before and after a transition in a virtual space. [Figure 13] FIG. 10 is a diagram illustrating an icon for displaying a 360-degree video. [Figure 14] 10A and 10B are diagrams illustrating the display forms of icons before and after a transition from the world of a 3DCG model image to the world of a 360-degree image. [Figure 15] FIG. 15 is a diagram illustrating the continuation of FIG. 14. [Figure 16] FIG. 10 is a diagram illustrating an example of a display image. [Figure 17] FIG. 17 is a diagram illustrating the continuation of FIG. 16. [Figure 18] FIG. 18 is a diagram illustrating the continuation of FIG. 17. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. An overview of the present invention will be described. In a virtual space in which a 3DCG model created by photogrammetry is placed, an icon of the 360-degree video is placed at a position corresponding to the shooting position of the 360-degree video. Then, in the virtual space, when a user approaches the icon, the present invention transitions from the world of the 3DCG model video (hereinafter referred to as "3DCG model video") to the world of the 360-degree video, and the 360-degree video is gradually enlarged and displayed. For example, when a user performs a key operation, the present invention reduces the enlarged 360-degree video and transitions from the world of the 360-degree video to the world of the original 3DCG model video.

[0028] This allows users to experience 6DoF in the world of 3DCG model images, and 3DoF in the world of 360-degree images.

[0029] In this way, by presenting the 3DCG model video and the 360-degree video in a complementary manner, it is possible to compensate for the drawback of parts of the 3DCG model video being missing, and to geometrically match the two videos and achieve visual continuity. In other words, a user viewing the 3DCG model video and the 360-degree video in a virtual space can correctly understand the two videos in a complementary manner.

[0030] [Summary of the Invention] First, we will explain the outline of the present invention. In order for a user to correctly understand the complementary relationship between a 3DCG model image and a 360-degree image in a virtual space, it is necessary to maintain visual continuity between these two images, i.e., to seamlessly connect the two images.

[0031] Specifically, when a user approaches an icon, the world transitions from a 3DCG model image to a 360-degree image, and when, for example, a key is pressed, the world transitions from a 360-degree image to a 3DCG model image, and it is necessary to achieve visual continuity between the two images.

[0032] The following conditions (A) to (D) can be considered to achieve visual continuity between the two videos. (A) The two worlds (the world of 3DCG model images and the world of 360-degree images) are geometrically identical. (B) Avoiding overlaps and gaps in time between the two worlds. (C) The two worlds change continuously. (D) The icon displaying the 360-degree video has an intuitive design.

[0033] <(A) The two worlds coincide geometrically> In order for the worlds to geometrically match before and after the transition from the world of the 3DCG model image to the world of the 360-degree image, the user must be present at a point in the 3DCG model image that appears the same as the shooting position of the 360-degree image before the transition.

[0034] Therefore, the video display processing device of the present invention starts a transition when the user's position (viewpoint position) in the world of the 3DCG model video approaches the shooting position of the 360-degree video. Note that the orientation of the 360-degree video is aligned with the orientation of the 3DCG model video.

[0035] Figure 12 is a diagram illustrating the state before and after a transition in a virtual space. Figure 12(1) shows a world of 3DCG model images, which includes images of a user 100 and two subjects (two buildings in this example). Figure 12(2) shows a world of 360-degree images, which also includes images of a user 100 and two subjects.

[0036] The position of user 100 in Fig. 12(1) is assumed to be at a position corresponding to the shooting position of the 360-degree video shown in Fig. 12(2). Also, the position of user 100 in Fig. 12(2) is assumed to be at the center of the sphere (screen sphere) of the spherical screen of the 360-degree video. In this case, the two worlds seen by user 100 will geometrically coincide.

[0037] <(B) Avoiding overlaps and gaps in time between the two worlds> The role of a transition is to replace the 3DCG model image and the 360-degree image. If the image is replaced too early or too late, the same subject may appear overlapping (doubled) in the two images, or both may be missing.

[0038] Therefore, when displaying two overlapping videos during a transition, the video display device of the present invention avoids overlapping and missing of the same subject by appropriately controlling presentation for each pixel in the field of view of the user 100. Furthermore, the video display processing device of the present invention avoids overlapping of the same subject by displaying the 360-degree video in the foreground.

[0039] <(C) The continuous change between the two worlds.> In order to reduce the cognitive cost of the two images for the user 100, it is necessary for the two worlds to change continuously.

[0040] Therefore, as in (A) above, the video display processing device of the present invention starts a transition when the position (viewpoint position) of user 100 in the world of the 3DCG model video approaches the shooting position of the 360-degree video. In this case, the video display processing device of the present invention does not immediately replace the entire 3DCG model video with the 360-degree video when the transition starts, but rather, before the transition starts, as user 100 approaches the icon of the 360-degree video, part of the field of view is gradually replaced with the icon of the 360-degree video. Furthermore, the transition also achieves a continuous change in appearance.

[0041] (D) The icon displaying the 360-degree video has an intuitive design. In order to clearly show the user 100 where and what kind of 360-degree images exist in the world of 3DCG model images, it is necessary to provide icons with affordances.

[0042] Therefore, the icon is placed in the world of the 3DCG model video at a position that corresponds to the shooting position of the 360-degree video, and is made to move slowly up and down, for example. Also, the icon is made to display a reduced image of the first frame of the 360-degree video as a thumbnail (icon image).

[0043] Fig. 13 is a diagram illustrating an icon that displays a 360-degree image. Fig. 13(1) shows the world of a 3DCG model image, and shows a user 100, an icon 101, two subjects, etc. Fig. 13(2) shows a display example of the icon 101, and Fig. 13(3) shows the rendering process of a spherical object.

[0044] 13(1) and 13(2), the shape of the icon 101 is a sphere, and a portion of the opening frame of the 360-degree video is pasted as a thumbnail on the inner surface of the sphere (icon sphere) of the icon 101. Furthermore, the icon 101 is placed at a position in the world of the 3DCG model video that corresponds to the shooting position of the 360-degree video.

[0045] As shown in Figure 13 (3), the spherical object for displaying the icon 101 is rendered using a culling function, targeting only the inside of the icon sphere. From the viewpoint of the user 100, α indicates the inner region of the spherical object's sphere that is rendered, and β indicates the outer region of the spherical object's sphere that is not rendered. In other words, when the icon 101 is displayed by rendering the spherical object, only the image of the region α, which is inside the spherical object's sphere, out of the still image that is the first frame of the 360-degree video, is displayed as a thumbnail of the icon 101.

[0046] Here, the icon 101 is displayed in a rotated state so that the visual orientation overlaps with the thumbnail and the 3DCG model image when viewed from the center of the sphere. In other words, when the viewpoint of the user 100 is at the center of the sphere of the icon 101, the icon 101 is displayed in a rotated state so that the subject included in the thumbnail displayed on the inner surface of the sphere and the same subject in the 3DCG model image are in the same direction when viewed from the viewpoint.

[0047] (Icon display format) Next, we will explain the display form of the icon 101. Fig. 14 is a diagram explaining the display form of the icon 101 before and after transition from the world of the 3DCG model image to the world of the 360-degree image, and Fig. 15 is a diagram explaining a continuation of Fig. 14.

[0048] In FIG. 14(1), an icon 101 and objects 102-1 and 102-2 exist in the world of a 3DCG model image as seen by a user 100.

[0049] Furthermore, the opening frame of the 360-degree video is pasted inside the icon 101, and the icon 101 is rendered by culling to target only the inside of the sphere (in this example, approximately a hemisphere) and exclude the outside of the sphere, so that only the portion seen from the viewpoint of the user 100 is displayed. As viewed from the user 100, only the image of the area α of the opening frame of the 360-degree video (out of the areas α and β inside the sphere) is displayed on the icon 101. The image of the area α includes an object 103-2 that corresponds to object 102-2 in the world of the 3DCG model video (the same subject is included). In other words, the object 102-2 in the world of the 3DCG model video and the object 103-2 included in the icon 101 are displayed so that their apparent orientations overlap when viewed from the viewpoint of the user 100.

[0050] The opening frame of the 360-degree video is displayed inside the sphere of icon 101, and the image of the α area inside the sphere as seen from the viewpoint of user 100 corresponds to the outside world, the 3DCG model video, projected onto the center of the sphere.

[0051] 14(2), the closer the user 100 gets to the icon 101, the more the display of the icon 101 seen by the user 100 and the display of the outside world (the display of the 3DCG model image) geometrically match. This is because, as the user 100 gets closer to the icon 101, the size of the sphere of the icon 101 becomes larger (appears larger) from the viewpoint of the user 100, the image inside the sphere also appears larger, and the apparent orientation of the display of the icon 101 and the display of the 3DCG model image overlap more closely.

[0052] Also, as in Figure 14(1), icon 101 has had at least the inner hemisphere of the sphere rendered by culling, resulting in a larger portion being rendered than in Figure 14(1). As seen by user 100, icon 101 displays an image of region α, which is wider than that in Figure 14(1), from the opening frame of the 360-degree video. The image of region α includes objects 103-1 and 103-2, which correspond to objects 102-1 and 102-2 in the world of the 3DCG model video.

[0053] 14(3), as the user 100 approaches the icon 101 and enters the sphere of the icon 101 (as the distance between the user 100 and the icon 101 becomes smaller (shorter) than a predetermined threshold), a transition begins at that moment. As a result, the user 100 transitions from the world of the 3DCG model image to the world of the 360-degree image. In other words, a gradually expanding 360-degree image appears in the field of view of the user 100.

[0054] Specifically, when the video display processing device starts the transition, it treats the icon sphere, which is the sphere of icon 101, as a screen sphere, and applies the foreground material (displays the 360-degree video in the foreground) regardless of the distance between the viewpoint of user 100 and icon 101, and renders the spherical object so that the screen sphere expands, and starts processing to play and display the 360-degree video.

[0055] As a result, when the screen sphere expands, the distance from the viewpoint to the 360-degree image pasted inside the screen sphere increases, but by applying the foreground material to the 360-degree image, even if the 3DCG model image is physically in front of the 360-degree image (closer to the viewpoint), the 360-degree image will not be overwritten by the 3DCG model image, and the 360-degree image will be displayed in front of the 3DCG model image with priority.

[0056] In Fig. 15(4), the screen sphere of the icon 101 expands to the extent that the motion parallax of the spherical screen disappears. Then, when the expansion / reduction ratio reaches a predetermined expansion completion setting value, the expansion is completed as shown in Fig. 15(5).

[0057] In Figures 15(4) and (5), the foreground material is applied to the 360-degree video. Therefore, object 103-1 in the 360-degree video and object 102-1 in the 3DCG model video are not displayed overlapping each other, and object 103-2 in the 360-degree video and object 102-2 in the 3DCG model video are not displayed overlapping each other. Objects 103-1 and 103-2 in the 360-degree video are displayed in front of objects 102-1 and 102-2 in the 3DCG model video. Note that objects 102-1 and 102-2 in the 3DCG model video are shown in Figures 15(4) and (5) for illustrative purposes only, but are not actually displayed. Furthermore, as the screen sphere expands, the distortion of the 360-degree video as seen from the user 100's viewpoint gradually decreases.

[0058] (Image display example) Next, a specific example of a displayed image will be described. Fig. 16 is a diagram for explaining an example of a displayed image, and Figs. 17 and 18 are diagrams for explaining the continuation of that.

[0059] It is assumed that a user 100 is having a 6DoF virtual experience in the world of a 3DCG model image.

[0060] Referring to Figure 16(1), user 100 can see a spherical icon 101 floating in the world of a 3DCG model video. On icon 101, a thumbnail of a reduced image of the world beyond icon 101 is displayed, that is, only the image visible from the user's 100's viewpoint from the opening frame of the 360-degree video is displayed as a thumbnail rendered by culling. Figure 16(1) corresponds to Figure 14(1).

[0061] Referring to FIG. 16(2), when the user 100 operates the controller, the viewpoint position and line of sight of the user 100 change, and the display of the thumbnail of the icon 101 also changes (the display as seen from the viewpoint changes). Then, as the user 100 approaches the icon 101, the display of the thumbnail of the icon 101 and the display of the world on the other side (the display of the 3DCG model image) geometrically coincide with each other. This is because the closer the user 100 is to the icon 101, the higher the proportion of the icon 101 occupies in the user's 100's field of view, making the icon 101 appear larger from the user's 100's viewpoint, and the image inside the icon sphere also appears larger, resulting in an even greater overlap between the apparent orientation of the display of the icon 101 and the display of the 3DCG model image. FIG. 16(2) corresponds to FIG. 14(2).

[0062] Assume that the user 100 approaches the icon 101 in accordance with the operation of the controller and enters the sphere of the icon 101.

[0063] Referring to Fig. 17(3), when the user 100 approaches the icon 101 in accordance with the operation of the controller, a thumbnail is displayed on the icon 101 until just before the user 100 enters the sphere of the icon 101. In addition, a 3DCG model image is displayed in areas other than the icon 101. Fig. 17(3) corresponds to Fig. 14(3).

[0064] When the user 100 enters the sphere of the icon 101, a transition starts at that moment, the icon sphere is treated as a screen sphere, the screen sphere starts expanding, and the 360-degree video is displayed enlarged on the screen sphere while being played back. As a result, the user 100 transitions from the world of the 3DCG model video to the world of the 360-degree video.

[0065] That is, referring to Figures 17(4) and 18(5), the thumbnails of the icon sphere are displayed so as to be played as a continuous 360-degree video, and the icon sphere becomes a screen sphere on which the 360-degree video is played. Figures 17(4) and 18(5) correspond to Figures 15(4) and 15(5).

[0066] This allows the user 100 to move from the world of the 3DCG model image to the world of the 360-degree image by approaching the icon 101 placed near the subject of the 3DCG model image, and for example, to recognize details of the subject in the 360-degree image.

[0067] Here, when the transition begins, the foreground material is applied to the 360-degree video, so even if the screen sphere expands beyond the distance at which the 3DCG model video exists inside the 360-degree video (closer to the viewpoint) based on the viewpoint, the 3DCG model video will not be displayed. In other words, even if the subject of the 3DCG model video exists inside the screen sphere as a result of the screen sphere expanding, the 360-degree video will be displayed with priority over the 3DCG model video.

[0068] Then, when the scaling ratio of the screen sphere reaches a predetermined scaling completion setting value, the scaling of the screen sphere is completed. After that, for example, in accordance with a key operation by the user 100, the screen sphere shrinks and the 3DCG model image is displayed, and when the scaling ratio of the screen sphere becomes smaller than the predetermined scaling completion setting value and returns to the original size, the application of the foreground material to the 360-degree image (application of the foreground display) is canceled, and the reduction is completed.

[0069] When the user 100 operates the controller to move away from the icon 101 and go outside the sphere of the icon 101 (when the distance exceeds a predetermined threshold), the user returns to the world of the 3DCG model image as shown in Figures 14(1) and (2) and Figures 16(1) and (2).

[0070] [Video display processing device] Next, a video display processing device according to an embodiment of the present invention will be described below. Fig. 1 is a block diagram showing an example of the configuration of a video display processing device according to an embodiment of the present invention.

[0071] This video display processing device 1 includes transition function units 2-1, 2-2, . . . , 2-N, 3DCG model objects consisting of spherical objects 3-1, 3-2, . . . , 3-N, 3DCG objects 4-1, 4-2, . . . , 4-M, etc., a camera object 5, a renderer 6, and a display device 7. N and M are integers equal to or greater than 2.

[0072] The video display processing device 1 places spherical objects 3-1, 3-2, ..., 3-N in a virtual space in which 3DCG model objects consisting of 3DCG objects 4-1, 4-2, ..., 4-M, etc. created by photogrammetry are placed, and displays 3DCG model images of the 3DCG model objects that are within the field of view of the user 100, as well as displays 360-degree images of the spherical objects 3-1, 3-2, ..., 3-N that are within the field of view of the user 100 as icons 101.

[0073] Each of the spherical objects 3-1, 3-2, ..., 3-N is configured with data related to an icon 101 on which a 360-degree video is displayed, and is attached to a corresponding transition function unit 2-1, 2-2, ..., 2-N. In other words, the data of each of the spherical objects 3-1, 3-2, ..., 3-N is basically set by the corresponding transition function unit 2-1, 2-2, ..., 2-N.

[0074] The 3DCG model object made up of 3DCG objects 4-1, 4-2, . . . , 4-M, etc. is made up of data relating to objects, etc., which are components when a 3DCG model image is displayed.

[0075] Hereinafter, the transition function units 2-1, 2-2, . . . , 2-N will be collectively referred to as the transition function unit 2, the spherical objects 3-1, 3-2, . . . , 3-N will be collectively referred to as the spherical object 3, and the 3DCG objects 4-1, 4-2, . . . , 4-M will be collectively referred to as the 3DCG object 4.

[0076] The transition function unit 2 has a corresponding spherical object 3 attached to it, and manages the state of the transition when transitioning from the world of 3DCG model images to the world of 360-degree images, or vice versa, in the virtual space world, and sets data in the spherical object 3 according to the state of the transition.

[0077] Specifically, depending on the state of the transition, the transition function unit 2 sets the opening frame of the 360-degree video as a still image, sets a spherical image representing the still image as the spherical object 3, plays the 360-degree video, sets this as the played 360-degree video, and sets a spherical image representing the played 360-degree video as the spherical object 3.

[0078] Furthermore, the transition function unit 2 calculates the scaling factor scale_magni(t) of the screen sphere on which the 360-degree video is displayed according to the state of the transition, and sets this as the size of the spherical object 3 in the spherical object 3. The transition function unit 2 also changes the material (unlit material or foreground material) and sets it in the spherical object 3. The t in the scaling factor scale_magni(t) indicates time.

[0079] In addition, the transition function unit 2 determines whether or not all objects other than the spherical object 3 (other spherical objects 3 and 3DCG objects 4) are to be displayed depending on the state of the transition, and sets displayability information indicating whether or not to display the other spherical objects 3 and 3DCG objects 4.

[0080] Here, the transition function unit 2 initially stores a list of other spherical objects 3, 3DCG objects 4, and camera objects 5, excluding the attached spherical object 3. The transition function unit 2 can acquire the positions, directions, etc. set for the other spherical objects 3, 3DCG objects 4, and camera object 5, and can also change data such as display availability information set for these objects. A detailed description of the processing of the transition function unit 2 will be given later.

[0081] The spherical objects 3 are attached to the corresponding transition function units 2, and various data output from the transition function units 2 is set to the spherical objects 3. As an initial setting, the spherical objects 3 are placed in a position in the virtual space that corresponds to the shooting position of the 360° video. Multiple spherical objects 3 are placed at a distance from each other so that they do not overlap.

[0082] 2 is a diagram showing an example of the data configuration of a spherical object 3. The spherical object 3 holds various types of data shown in FIG. 2. The various types of data constituting the spherical object 3 include a spherical object position indicating the position where the spherical object 3 is placed in the virtual space (a spherical object position in which the shooting position of the 360-degree video of the spherical object 3 is the placement position of the spherical object 3, and the center position of the spherical object 3), a direction in which the spherical object 3 is facing, a size of the video when the spherical object 3 is displayed (scaling ratio scale_magni(t)), a spherical video indicating the video in which the spherical object 3 is displayed (a spherical video showing a still image or a spherical video showing a played 360-degree video), a material (an unlit material or a foreground material), display availability information indicating whether the object is hidden or displayed, audio information, and movement information.

[0083] The 3DCG object 4 is a 3DCG model created by photogrammetry, and data relating to the object in the world of the 3DCG model image is set in the 3DCG object 4. The 3DCG object 4 is initially placed at a predetermined position in the virtual space.

[0084] The data set for the 3DCG object 4 includes, for example, the position and orientation of the object in the virtual world, the size, shape, and image of the object, and the displayability information mentioned above. The size of the object is set to, for example, the actual size.

[0085] Incidentally, for the spherical object 3 and the 3DCG object 4, a single object or a plurality of objects may be grouped, and the display availability information and the like may be handled collectively for each group.

[0086] The camera object 5 is set with data such as the viewpoint position and line of sight direction of the user 100 in the virtual space.

[0087] Here, the camera processing unit (not shown) determines the position and direction of the virtual camera in the virtual space as the viewpoint position and line of sight direction of the user 100 in accordance with the operation of the user 100 on the controller (not shown), and sets this data in the camera object 5.

[0088] The renderer 6 identifies the 3DCG object 4 photographed by the virtual camera in the virtual space, draws (renders) a 3DCG model object including the identified 3DCG object 4, and also identifies and renders the spherical object 3 photographed by the virtual camera, generates a display image, and outputs it to the display device 7.

[0089] Specifically, based on the viewpoint position and line of sight direction of the user 100 (position and direction of the virtual camera) set in the camera object 5, the renderer 6 identifies the spherical object 3 and the 3DCG object 4 within the range captured by the virtual camera (within the field of view of the user 100) using data such as the positions set in these objects.

[0090] The renderer 6 then renders the identified 3DCG object 4 pixel by pixel using data such as displayability information set for the 3DCG object 4, and also renders the identified spherical object 3 pixel by pixel using data such as size, spherical image, material, and displayability information set for the spherical object 3. The renderer 6 then generates a display image by combining both images obtained by rendering and outputs the image to the display device 7.

[0091] If the displayability information for the spherical object 3 indicates that it should not be displayed, the renderer 6 does not render the spherical object 3, but if the displayability information indicates that it should be displayed, the renderer 6 renders the spherical object 3. The same applies to the 3DCG object 4.

[0092] Furthermore, when rendering the spherical object 3, the renderer 6 pastes the spherical image set in the spherical object 3 onto the inside of the sphere, and enlarges or reduces the image of the sphere as seen from the viewpoint of the user 100 according to the viewpoint position and line of sight set in the camera object 5 and the size set in the spherical object 3, thereby generating a display image including an image of the icon sphere or the screen sphere.

[0093] Furthermore, when rendering a spherical object 3, if the material set for the spherical object 3 is the foreground material, the renderer 6 generates a display image so that the spherical image of the spherical object 3 is displayed in the foreground in preference to the images of other spherical objects 3 and 3DCG objects 4.

[0094] The display device 7 is, for example, an HMD (Head Mounted Display), which receives the display image from the renderer 6 and displays the display image on a screen.

[0095] This generates a display image (for example, an image shown in Figure 16(1) or (2), Figure 17(3) or (4), or Figure 18(5)) that corresponds to the viewpoint position and line of sight of the user 100, and displays it on the display device 7.

[0096] The processing of the renderer 6 is realized by the CG drawing processing of a general game engine such as Unity.

[0097] [Transition function section 2] Next, a detailed description will be given of the transition function unit 2 shown in Fig. 1. Fig. 3 is a block diagram showing an example of the configuration of the transition function unit 2.

[0098] This transition function unit 2 includes a distance calculation unit 10, a key input unit 11, a state management unit 12, a zoom unit 13, a material recording unit 14, a material change unit 15, a 360-degree video recording unit 16, a video playback control unit 17, a still image extraction unit 18, a video selection unit 19, and an other object display control unit 20. The 360-degree video recording unit 16, the video playback control unit 17, the still image extraction unit 18, and the video selection unit 19 constitute a video playback processing unit.

[0099] The distance calculation unit 10 receives as input the viewpoint position set in the camera object 5, and also receives as input the spherical object position set in the spherical object 3 corresponding to the transition function unit 2.

[0100] The distance calculation unit 10 calculates the distance distance(t) between the viewpoint position of the user 100 and the position of the spherical object 3 (the center position of the icon sphere or screen sphere) by using the coordinates of the viewpoint position and the coordinates of the spherical object position to calculate the root of the sum of squares of the difference between these positions. The t in distance distance(t) represents time. The distance calculation unit 10 outputs the distance distance(t) to the state management unit 12 as distance information.

[0101] The key input unit 11 monitors input of key information from the controller in response to a key operation on the controller by the user 100. When the key input unit 11 inputs key information from the controller, it generates a key control signal corresponding to the key information and outputs the key control signal to the state management unit 12.

[0102] The state management unit 12 receives the distance distance(t) from the distance calculation unit 10, a key control signal from the key input unit 11, and the scaling factor scale_magni(t) of the screen sphere from the scaling unit 13. The state management unit 12 also receives the playback status of the 360-degree video (not played, playback completed, playing, paused, current playback time, etc.) from the video playback control unit 17.

[0103] The state management unit 12 sequentially transitions the transition states of the spherical object 3 (waiting S1, expanding S2, expanded S3, shrinking S4, and reduced S5) according to the conditions described below, as shown in the state transitions shown in Figure 4 described below.

[0104] Fig. 4 is a diagram showing state transitions of the spherical object 3. As shown in Fig. 4, there are five transition states of the spherical object 3: waiting S1, expanding S2, expanded S3, shrinking S4, and reduced S5.

[0105] As mentioned above, distance(t) is the distance between the viewpoint position of the user 100 and the position of the spherical object 3, and scale_magni(t) is the magnification ratio of the screen sphere, which is obtained by dividing the radius of the screen sphere by the radius of the preset icon sphere.

[0106] dist_th_in is a transition start threshold at which the screen sphere starts expanding, and is set in advance. As the transition start threshold dist_th_in, for example, the radius of the icon sphere is set in advance.

[0107] Furthermore, dist_th_out is a transition completion threshold for completing the transition and entering the standby S1 state, and is set in advance. A value greater than the transition start threshold dist_th_in, for example, a value twice the radius of the icon sphere, is set in advance as the transition completion threshold dist_th_out. The reason why a value greater than the transition start threshold dist_th_in is set as the transition completion threshold dist_th_out is to prevent chattering of an unintended state when the distance distance(t) approaches the transition start threshold dist_th_in and the transition completion threshold dist_th_out.

[0108] The max_scale_magni is a magnification completion threshold value that indicates the magnification / reduction ratio of the screen sphere at which the magnification of the screen sphere is completed, and is set in advance. For example, 50 is set as the magnification completion threshold value max_scale_magni.

[0109] When the transition state of the spherical object 3 is standby S1, if the condition of distance (t)<transition start threshold dist_th_in is satisfied, the transition state changes from standby S1 to expanding S2. This condition occurs when the user 100 gets closer to the spherical object 3 than the transition start threshold dist_th_in.

[0110] As long as the condition of distance (t)≧transition start threshold dist_th_in is satisfied, the transition state remains in the waiting state S1. This condition applies when the user 100 is farther away from the spherical object 3 than the transition start threshold dist_th_in.

[0111] When the transition state of the spherical object 3 is expanding S2, if the condition of the scaling rate scale_magni(t)≧the scaling completion threshold max_scale_magni is satisfied, the transition state changes from expanding S2 to expanded S3. This condition occurs when the scaling rate scale_magni(t), which is the size of the spherical object 3, reaches the scaling completion threshold max_scale_magni.

[0112] When the transition state of the spherical object 3 is enlarged S3, if a condition that a key input has been made by the user 100 is satisfied, the transition state changes from enlarged S3 to shrinking S4. The key input in this case is a key input for changing the transition state from enlarged S3 to shrinking S4.

[0113] Furthermore, when the transition state of the spherical object 3 is enlarged S3, if the condition that the playback of the 360-degree video has been completed (the condition that the state management unit 12 inputs the playback state of playback completion) is satisfied, the transition state may be changed from enlarged S3 to shrinking S4.

[0114] When the transition state of the spherical object 3 is shrinking S4 and the condition of the scaling factor scale_magni(t)<1.0 is met, the transition state changes from shrinking S4 to reduced S5. This condition occurs when the scaling factor scale_magni(t), which is the size of the spherical object 3, becomes smaller than 1.0.

[0115] When the transition state of the spherical object 3 is Shrunken S5, if the condition of distance (t)>transition completion threshold dist_th_out is satisfied, the transition state changes from Shrunken S5 to Waiting S1. This condition occurs when the user 100 is farther away from the spherical object 3 than the transition completion threshold dist_th_out.

[0116] It should be noted that if the display availability information of the spherical object 3 is set to non-display, the state transition shown in FIG. 4 does not occur.

[0117] Returning to Figure 3, the state management unit 12 generates a control signal according to the transition state of the spherical object 3, and outputs the control signal to the scaling unit 13, the material change unit 15, the video playback control unit 17, and the other object display control unit 20.

[0118] Specifically, the state management unit 12 generates a control signal indicating enlargement, stop of enlargement, reduction, or stop of reduction according to the transition state of the spherical object 3, and outputs the control signal to the scaling unit 13.

[0119] The state management unit 12 generates a control signal indicating the unlit material or the foreground material according to the transition state of the spherical object 3 , and outputs the control signal to the material change unit 15 .

[0120] The state management unit 12 generates a control signal indicating the start of the 360-degree video, the cue of the 360-degree video, or the end of the 360-degree video depending on the transition state of the spherical object 3, and outputs the control signal to the video playback control unit 17.

[0121] The state management unit 12 generates a control signal indicating whether to display or hide each of the objects other than the spherical object 3 (other spherical objects 3 and 3DCG objects 4) in accordance with the transition state of the spherical object 3. Then, the state management unit 12 outputs the control signal to the other object display control unit 20. The processing of the state management unit 12 will be described in detail later.

[0122] As an initial setting, the scaling unit 13 sets the scaling ratio scale_magni(t) obtained by dividing the radius of the screen sphere by the radius of a preset icon sphere to 1, outputs the scaling ratio scale_magni(t) to the state management unit 12, and sets the scaling ratio scale_magni(t) to the spherical object 3 as the size of the image when the spherical object 3 is displayed.

[0123] The enlargement / reduction unit 13 receives a control signal from the state management unit 12 indicating enlargement, stop of enlargement, reduction, or stop of reduction.

[0124] When the scaling unit 13 receives a control signal indicating enlargement, it calculates the scaling ratio scale_magni(t) by dividing the radius of the screen sphere by the radius of the preset icon sphere so that the screen sphere of the spherical object 3 enlarges (so that the radius of the screen sphere gradually increases over time). When the scaling unit 13 receives a control signal indicating stop of enlargement, it stops calculating the scaling ratio scale_magni(t) so that the enlargement of the screen sphere of the spherical object 3 stops (so that the process of increasing the radius of the screen sphere stops). As a result, the scaling ratio scale_magni(t) becomes a constant value when enlargement stops.

[0125] When the scaling unit 13 receives a control signal indicating reduction, it calculates the scaling rate scale_magni(t) so that the spherical object 3 reduces (so that the radius of the screen sphere gradually decreases over time). When the scaling unit 13 receives a control signal indicating stop of reduction, it stops calculating the scaling rate scale_magni(t) so that the reduction of the screen sphere of the spherical object 3 stops (so that the process of reducing the radius of the screen sphere stops). As a result, the scaling rate scale_magni(t) becomes a constant value when reduction stops.

[0126] Furthermore, the scaling unit 13 outputs the calculated scaling factor scale_magni(t) to the state management unit 12, and sets the scaling factor scale_magni(t) to the spherical object 3 as the size of the spherical object 3. As a result, the size of the spherical object 3 is set to the spherical object 3.

[0127] The material recording section 14 stores unlit material and foreground material.

[0128] A material is a program and a set of setting data that define the rendering rules for CG objects, including spherical surfaces, in a virtual space. An unlit material is a program that defines general rendering rules when lighting is disabled (when there is no lighting). A foreground material is a program that defines rendering rules when a particular image (in an embodiment of the present invention, a 360-degree playback image) is displayed in the foreground of all images so that the image is always displayed regardless of the distance from the viewpoint of the user 100 and other images are not overwritten on the image.

[0129] As an initial setting, the material change unit 15 reads out an unlit material from the material recording unit 14 and sets the unlit material to the spherical object 3 .

[0130] The material change unit 15 receives a control signal from the state management unit 12 indicating the unlit material or the foreground material.

[0131] When the material change unit 15 receives a control signal indicating an unlit material, it changes the material to the unlit material by reading out the unlit material from the material recording unit 14. On the other hand, when the material change unit 15 receives a control signal indicating a foreground material, it reads out the foreground material from the material recording unit 14 and changes the material to the foreground material.

[0132] The material change unit 15 sets the changed unlit material or foreground material to the spherical object 3. As a result, the unlit material or the foreground material is set to the spherical object 3.

[0133] The 360-degree video recording unit 16 records a 360-degree video of the spherical object 3.

[0134] As an initial setting, the video playback control unit 17 reads the 360-degree video from the 360-degree video recording unit 16, starts playback of the 360-degree video from the beginning, pauses playback at the opening frame to cue, sets the playback time to the time of the opening frame, and sets the playback status of the 360-degree video to paused.The video playback control unit 17 then outputs the paused playback status to the status management unit 12. In this case, when the status management unit 12 receives the paused playback status from the video playback control unit 17 as an initial setting, it sets the transition status to standby S1.

[0135] Furthermore, the video playback control unit 17 outputs the 360-degree video to the still image extraction unit 18. As a result, the still image extraction unit 18 extracts the opening frame from the 360-degree video as a still image.

[0136] The video playback control unit 17 receives a control signal from the state management unit 12 that indicates the start of a 360-degree video, cueing of a 360-degree video, or the end of a 360-degree video.

[0137] When the control signal indicating playback of the 360-degree video is input, the video playback control unit 17 plays the 360-degree video that was paused at the beginning frame and changes the playback status of the 360-degree video to "playing." The video playback control unit 17 then outputs the playback status of "playing" to the status management unit 12. The video playback control unit 17 also outputs the played 360-degree video to the video selection unit 19 as "played 360-degree video."

[0138] When the control signal instructing to cue the 360-degree video is input, the video playback control unit 17 starts playing the 360-degree video from the beginning, pauses the playback at the opening frame to cue it, sets the playback time to the time of the opening frame, and sets the playback status of the 360-degree video to "paused." The video playback control unit 17 then outputs the paused playback status to the status management unit 12, and outputs the opening frame of the 360-degree video to the video selection unit 19 as the opening frame of the played 360-degree video.

[0139] When the video playback control unit 17 receives a control signal indicating the end of the 360-degree video, it prevents the playback 360-degree video from being output to the video selection unit 19. This causes the video selection unit 19 to receive only the still images from the still image extraction unit 18.

[0140] When playback of the 360-degree video is complete, the video playback control unit 17 sets the playback status of the 360-degree video to "playback completed" and outputs the playback status of "playback completed" to the state management unit 12. Furthermore, when the 360-degree video has not yet been played, the video playback control unit 17 sets the playback status of the 360-degree video to "not yet played" and outputs the playback status of "not yet played" to the state management unit 12. Furthermore, the video playback control unit 17 outputs the playback status, such as the current playback time, to the state management unit 12.

[0141] Still image extraction unit 18 receives the 360-degree video from video playback control unit 17, extracts the opening frame from the 360-degree video, and outputs the opening frame as a still image to video selection unit 19. Even if the 360-degree video is no longer being input to still image extraction unit 18, the output state of the still image is maintained. This ensures that the video selection unit 19 is always receiving the still image.

[0142] Video selection unit 19 receives as input still images from still image extraction unit 18 and receives as input reproduced 360-degree video from video reproduction control unit 17 .

[0143] When a still image is input but a 360-degree playback image (including the opening frame of the 360-degree playback image) is not input, the image selection unit 19 selects a spherical image representing the still image and sets it as the spherical object 3. As a result, when the user 100 views the spherical object 3, the spherical object 3 functions as an icon sphere. In other words, when the renderer 6 renders the spherical object 3, a thumbnail of the still image is displayed in the icon sphere (inside the sphere of the icon 101).

[0144] When a still image is input and a replayed 360-degree video is input, the video selection unit 19 selects a spherical video representing the replayed 360-degree video and sets it as the spherical object 3. As a result, when the user 100 views the spherical object 3, the spherical object 3 functions as a screen sphere. In other words, the renderer 6 renders the spherical object 3, and the replayed 360-degree video (the replayed 360-degree video) is displayed on the screen sphere.

[0145] The other object display control unit 20 sets, as an initial setting, displayability information indicating display for each of the objects (other spherical objects 3 and 3DCG objects 4) other than the spherical object 3. As a result, displayability information indicating display is set, as an initial setting, for each of all spherical objects 3 and 3DCG objects 4.

[0146] The other object display control unit 20 receives a control signal from the state management unit 12 that indicates whether to display or hide each of the objects other than the spherical object 3 (other spherical objects 3 and 3DCG objects 4).

[0147] When the other object display control unit 20 receives a control signal indicating that an object other than the spherical object 3 is to be displayed, it sets displayability information indicating that the object is to be displayed to the object other than the spherical object 3. When the other object display control unit 20 receives a control signal indicating that an object other than the spherical object 3 is not to be displayed, it sets displayability information indicating that the object is not to be displayed to the object other than the spherical object 3.

[0148] (Processing example of the state management unit 12) Next, a detailed description will be given of the processing of the state management unit 12 shown in Fig. 3. Fig. 5 is a flowchart showing an example of the processing of the state management unit 12.

[0149] When the state management unit 12 receives the playback state of paused as an initial setting from the video playback control unit 17, it sets the transition state SS to standby S1 (step S501). In this case, it is assumed that the distance distance(t) between the viewpoint position of the user 100 and the position of the spherical object 3 received from the distance calculation unit 10 is greater (longer) than a preset transition start threshold dist_th_in.

[0150] As shown in FIGS. 14(1) and (2) and 16(1) and (2), this waiting S1 state SS indicates that the user 100 is away from the icon 101 by the transition start threshold dist_th_in or more.

[0151] When the transition state SS is standby S1, the size of the spherical object 3 with a scaling factor scale_magni(t)=1.0, a spherical image representing a still image, unlit material, display availability information indicating whether to display, etc. are set for the spherical object 3. The same applies to the other spherical objects 3. Furthermore, the 3DCG object 4 is set with display availability information indicating whether to display, etc.

[0152] Here, the renderer 6 identifies the spherical object 3 and the like within the range captured by the virtual camera based on the position and viewpoint direction of the user 100 set in the camera object 5. Then, the renderer 6 pastes a still image onto the inside of the spherical surface of the identified spherical object 3, and performs rendering at the viewpoint position and line of sight of the user 100, excluding the outside of the sphere by culling and targeting only the inside, thereby generating an image of the icon 101, which is a thumbnail of the icon sphere (with the still image pasted inside the icon sphere) as seen from the viewpoint of the user 100.

[0153] The renderer 6 also generates a 3DCG model image by rendering 3DCG model objects including the 3DCG object 4 within the range photographed by the virtual camera. The renderer 6 then generates a display image by combining the image of the icon 101, which is a thumbnail of the icon sphere, and the 3DCG model image, and outputs the display image to the display device 7.

[0154] As a result, a 3DCG model image including a thumbnail icon 101 is displayed, as shown in FIGS. 14(1) and (2) and FIGS. 16(1) and (2).

[0155] The state management unit 12 determines whether the distance distance(t) between the viewpoint position of the user 100 and the position of the spherical object 3 input from the distance calculation unit 10 is smaller (shorter) than a preset transition start threshold value dist_th_in (step S502, distance(t) <dist_th_in)。

[0156] If it is determined in step S502 that the distance distance(t) is not smaller than the transition start threshold dist_th_in (step S502: N), the state management unit 12 proceeds to step S502 after a certain time has elapsed (t←t+1) (step S503).

[0157] If the state management unit 12 determines in step S502 that the distance distance(t) is smaller than the transition start threshold dist_th_in (step S502: Y), it sets the transition state SS to expanding S2 (step S504) and performs expanding processing (step S505). This starts the transition, and the screen sphere begins to expand. The screen sphere gradually becomes larger from the same size as the icon sphere. In other words, it is possible to achieve visual continuity when transitioning from the world of 3DCG model images to the world of 360-degree images.

[0158] In this expanding S2 state SS, as shown in Fig. 14(3), Fig. 15(4) and (5), Fig. 17(3) and (4), and Fig. 18(5), the user 100 is closer to the icon 101 than the transition start threshold dist_th_in. In the expanded S3, reducing S4, and reduced S5 states SS, which will be described later, the user 100 is also closer to the icon 101 than the transition start threshold dist_th_in.

[0159] 6 is a flowchart showing an example of the enlargement process (step S505). When the process proceeds from step S504 to step S505 shown in FIG. 5, the state management unit 12 generates a control signal indicating the start of the 360-degree video in the enlargement process and outputs the control signal to the video playback control unit 17 (step S601).

[0160] As a result, the video playback control unit 17 plays back the 360-degree video that was paused at the beginning frame, and the video selection unit 19 sets a spherical video representing the played back 360-degree video in the spherical object 3.

[0161] The state management unit 12 generates a control signal indicating the foreground material, and outputs the control signal to the material change unit 15 (step S602).

[0162] As a result, the material change unit 15 changes the unlit material to the foreground material, and the foreground material is set for the spherical object 3.

[0163] The state management unit 12 generates a control signal indicating enlargement, and outputs the control signal to the enlargement / reduction unit 13 (step S603).

[0164] As a result, the scaling unit 13 calculates the scaling ratio scale_magni(t) so that the radius of the screen sphere of the spherical object 3 gradually increases over time, and sets this as the size of the spherical object 3.

[0165] The state management unit 12 generates a control signal indicating non-display for each of the other objects other than the spherical object 3 (other spherical objects 3 and 3DCG objects 4), and outputs the control signal to the other object display control unit 20 (step S604).

[0166] As a result, the other object display control unit 20 sets the display availability information indicating non-display for the other object.

[0167] In this way, when the transition of the spherical object 3 is in the expanding state S2, the above data is set for the spherical object 3 and other objects. Then, the renderer 6 renders only the spherical object 3 for which display availability information indicating display has been set, and a gradually expanding reproduced 360-degree video is displayed on the display device 7, as shown in Figures 14(3), 15(4) and (5), 17(3) and (4), and 18(5).

[0168] Here, the renderer 6 pastes the reproduced 360-degree video onto the inside of the spherical surface of the spherical object 3 and renders it at the viewpoint position and line of sight of the user 100, thereby generating an image of a screen sphere that grows over time according to the size set for the spherical object 3 and is displayed in the foreground according to the foreground material set for the spherical object 3.

[0169] Furthermore, the renderer 6 does not perform rendering for the spherical objects 3 other than the spherical object 3 in question within the range photographed by the virtual camera, because the display availability information indicating non-display is set for these spherical objects 3. The same applies to the 3DCG objects 4 within the range photographed by the virtual camera.

[0170] The renderer 6 generates a display image that includes the image of the screen sphere but does not include images of spherical objects 3 and 3DCG objects 4 other than the spherical object 3 within the range photographed by the virtual camera, and displays the image of the screen sphere in the foreground, and outputs the generated image to the display device 7.

[0171] In other words, because the foreground material is set for the spherical object 3, the played 360-degree video is displayed in the foreground, and because display availability information indicating non-display is set for the 3DCG object 4, the video of the 3DCG object 4 is not displayed. This makes it possible to compensate for the drawback of parts of the 3DCG model video being missing or distorted.

[0172] In addition, in the video playback control unit 17, one of loop setting, stop setting, and end setting is set in advance as the operation mode when playback of the 360-degree video is completed.

[0173] If loop setting is set as the operation mode, when playback of the 360-degree video is completed, the video playback control unit 17 sets the playback time to the time of the first frame, continues playback from the first frame of the 360-degree video, and continues outputting the played 360-degree video to the video selection unit 19. As a result, rendering returns to the first frame of the played 360-degree video, and it is displayed on the display device 7.

[0174] Furthermore, if the operation mode is set to the stop setting, when playback of the 360-degree video is completed, the video playback control unit 17 pauses the playback at the final frame and outputs the final frame of the played back 360-degree video to the video selection unit 19. This causes the final frame of the played back 360-degree video to be rendered and displayed on the display device 7.

[0175] Furthermore, if the operation mode is set to end setting, when playback of the 360-degree video is completed, the video playback control unit 17 outputs the playback status of playback complete to the status management unit 12. When the status management unit 12 receives the playback status of playback complete from the video playback control unit 17, it proceeds to step S512 (described later) without performing the processes of steps S510 and S511 (described later), and sets the transition status SS to shrinking S4. These processes are also performed when the status SS is enlarged S3.

[0176] Returning to Figure 5, the state management unit 12 transitions from the enlargement processing in step S505 and determines whether the enlargement / reduction ratio scale_magni(t) input from the enlargement / reduction unit 13 is greater than or equal to the preset enlargement completion threshold max_scale_magni (step S506, scale_magni(t) ≧ max_scale_magni).

[0177] If it is determined in step S506 that the enlargement / reduction rate scale_magni(t) is smaller than the enlargement completion threshold max_scale_magni (step S506: N), the state management unit 12 proceeds to step S506 after a certain time has elapsed (t←t+1) (step S507).

[0178] If it is determined in step S506 that the enlargement / reduction rate scale_magni(t) is equal to or greater than the enlargement completion threshold max_scale_magni (step S506: Y), the state management unit 12 sets the transition state SS to enlarged S3 (step S508) and performs enlargement processing (step S509), thereby completing the enlargement of the screen sphere.

[0179] 7 is a flowchart showing an example of the enlargement processing (step S509). When the process proceeds from step S508 to step S509 shown in FIG. 5, the state management unit 12 generates a control signal indicating stop of enlargement in the enlargement processing and outputs the control signal to the enlargement / reduction unit 13 (step S701).

[0180] As a result, the process of gradually increasing the radius of the screen sphere of the spherical object 3 over time is stopped in the scaling unit 13, and the scaling factor scale_magni(t) becomes a constant value. Then, this constant value of the scaling factor scale_magni(t) is set for the spherical object 3 as the size of the spherical object 3.

[0181] In this way, when the transition of the spherical object 3 is in the enlarged state S3 SS, a fixed value of the scaling factor scale_magni(t) is set to the spherical object 3 as the size of the spherical object 3, and the renderer 6 displays the enlarged reproduced 360-degree image on the display device 7.

[0182] Here, the renderer 6 pastes the reproduced 360-degree image onto the inside of the spherical surface of the spherical object 3 and renders it at the viewpoint position and line of sight of the user 100, thereby generating an image of a screen sphere that has a fixed size set for the spherical object 3 and is displayed in the foreground according to the foreground material set for the spherical object 3.

[0183] Furthermore, the renderer 6 does not perform rendering for the spherical objects 3 other than the spherical object 3 in question within the range photographed by the virtual camera, because the display availability information indicating non-display is set for these spherical objects 3. The same applies to the 3DCG objects 4 within the range photographed by the virtual camera.

[0184] The renderer 6 generates a display image that includes the image of the screen sphere after enlargement, but does not include images of spherical objects 3 and 3DCG objects 4 other than the spherical object 3 within the range photographed by the virtual camera, and displays the image of the screen sphere in the foreground, and outputs the generated image to the display device 7.

[0185] In other words, just as in the case of the expanding S2 state SS, the foreground material is set for the spherical object 3, so the played 360-degree video is displayed in the foreground, and the display availability information indicating that the 3DCG object 4 is hidden is set, so the video of the 3DCG object 4 is not displayed. This makes it possible to compensate for the drawback of parts of the 3DCG model video being missing or distorted.

[0186] 5, the state management unit 12 moves on from the enlarged processing in step S509 and determines whether or not a key control signal has been input from the key input unit 11 (step S510). This key control signal is a signal output from the key input unit 11 when the user 100 performs a key operation using the controller to transition the state SS from Enlarged S3 to Reduced S4.

[0187] If it is determined in step S510 that a key control signal has not been input (step S510: N), the state management unit 12 proceeds to step S510 after a certain time has elapsed (t←t+1) (step S511).

[0188] If it is determined in step S510 that a key control signal has been input (step S510: Y), the state management unit 12 sets the transition state SS to shrinking S4 (step S512) and performs shrinking processing (step S513). This starts shrinking the screen sphere.

[0189] 8 is a flowchart showing an example of the reduction-in-progress process (step S513). When the process proceeds from step S512 to step S513 shown in FIG. 5, the state management unit 12 generates a control signal instructing the cueing of the 360-degree video in the reduction-in-progress process, and outputs the control signal to the video playback control unit 17 (step S801).

[0190] As a result, the video playback control unit 17 starts playing the 360-degree video from the beginning, pauses it at the opening frame, and then the video selection unit 19 sets the spherical video of the opening frame of the played 360-degree video to the spherical object 3.

[0191] The state management unit 12 generates a control signal indicating reduction, and outputs the control signal to the scaling unit 13 (step S802).

[0192] As a result, the scaling unit 13 calculates the scaling ratio scale_magni(t) so that the radius of the screen sphere of the spherical object 3 gradually becomes smaller over time, and sets this as the size of the spherical object 3.

[0193] The state management unit 12 generates a control signal indicating display for each of the other objects other than the spherical object 3 (other spherical objects 3 and 3DCG objects 4), and outputs the control signal to the other object display control unit 20 (step S803).

[0194] As a result, the other object display control unit 20 sets display availability information indicating whether or not to display the other object.

[0195] In this way, when the transition of the spherical object 3 is in the shrinking state S4 SS, the above-mentioned data is set for the spherical object 3 and other objects. Then, rendering is performed by the renderer 6 for the spherical object 3 and other objects within the range captured by the virtual camera, which is specified by the viewpoint position and line of sight of the user 100 set in the camera object 5, and the opening frame of the gradually shrinking reproduced 360-degree video is displayed on the display device 7, and the video of the other objects is redisplayed on the display device 7.

[0196] Here, the renderer 6 pastes the opening frame of the reproduced 360-degree video onto the inside of the spherical surface of the spherical object 3 and renders it at the viewpoint position and line of sight of the user 100, thereby generating an image of a screen sphere that becomes smaller over time according to the size set for the spherical object 3 and is displayed in the foreground according to the foreground material set for the spherical object 3.

[0197] Furthermore, because display availability information indicating display is set for spherical objects 3 other than the spherical object 3 within the range photographed by the virtual camera, the renderer 6 performs rendering and generates an image of the spherical object 3. The renderer 6 performs similar processing on the 3DCG objects 4 within the range photographed by the virtual camera and generates an image of the 3DCG objects 4.

[0198] The renderer 6 generates a display image in which the image of the screen sphere is displayed in the foreground, including images of the spherical object 3 other than the spherical object 3 in question within the range photographed by the virtual camera, and images of the 3DCG object 4, and outputs this to the display device 7. In other words, as in the cases of the expanding S2 and expanded S3 states SS, the foreground material is set for the spherical object 3 in question, so the played 360-degree image is displayed in the foreground.

[0199] Returning to FIG. 5, the state management unit 12 proceeds from the reduction processing in step S513 and determines whether the scaling ratio scale_magni(t) input from the scaling unit 13 is smaller than the preset value 1.0 (step S514, scale_magni(t)<1.0).

[0200] Note that instead of the preset value 1.0, a preset reduction completion threshold min_scale_magni other than 1.0 may be used. In this case, in step S514, the state management unit 12 determines whether the enlargement / reduction ratio scale_magni(t) is smaller than the preset reduction completion threshold min_scale_magni.

[0201] If it is determined in step S514 that the enlargement / reduction ratio scale_magni(t) is not smaller than 1.0 (step S514: N), the state management unit 12 proceeds to step S514 after a certain time has elapsed (t←t+1) (step S515).

[0202] If the state management unit 12 determines in step S514 that the scaling factor scale_magni(t) is smaller than 1.0 (step S514: Y), it sets the transition state SS to "reduced" S5 (step S516) and performs "reduced" processing (step S517). This completes the reduction of the screen sphere, and the transition state SS becomes approximately the same size as the icon sphere in standby S1. In other words, visual continuity can be achieved when transitioning from the 360-degree video world to the 3DCG model video world.

[0203] 9 is a flowchart showing an example of the reduction-completed process (step S517). When the process proceeds from step S516 to step S517 shown in FIG. 5, the state management unit 12 generates a control signal indicating that reduction is to be stopped in the reduction-completed process, and outputs the control signal to the scaling unit 13 (step S901).

[0204] As a result, the scaling unit 13 stops the process of gradually reducing the radius of the screen sphere of the spherical object 3 over time, and the scaling factor scale_magni(t) becomes a constant value smaller than 1.0. Then, this constant value smaller than 1.0, the scaling factor scale_magni(t), is set for the spherical object 3 as the size of the spherical object 3.

[0205] The state management unit 12 generates a control signal indicating the end of the 360-degree video, and outputs the control signal to the video playback control unit 17 (step S902).

[0206] As a result, the video playback control unit 17 stops outputting the opening frame of the played back 360-degree video (output ceases), and the video selection unit 19 sets the thumbnail spherical video to the spherical object 3.

[0207] The state management unit 12 generates a control signal indicating the unlit material, and outputs the control signal to the material change unit 15 (step S903).

[0208] As a result, the material change unit 15 changes the foreground material to the unlit material, and the unlit material from which the foreground material has been removed is set to the spherical object 3.

[0209] In this way, when the transition of the spherical object 3 is in the reduced state S5 SS, the display device 7 stops displaying the opening frame of the reproduced 360-degree video on the gradually reducing screen sphere, and when the radius of the screen sphere becomes smaller than that of the icon sphere, the screen sphere is replaced by an icon sphere and a thumbnail is displayed.

[0210] Here, the renderer 6 renders the spherical object 3 etc. within the range captured by the virtual camera by culling based on the position and viewpoint direction of the user 100 set in the camera object 5, thereby generating an image of the icon 101 (thumbnail spherical image) with a still image pasted inside the icon sphere as seen from the viewpoint of the user 100.

[0211] The renderer 6 also generates a 3DCG model image by rendering 3DCG model objects including the 3DCG object 4 within the range photographed by the virtual camera. The renderer 6 then combines the image of the icon 101 and the 3DCG model image to generate a display image, and outputs the display image to the display device 7.

[0212] Returning to Figure 5, the state management unit 12 transitions from the reduced processing in step S517 and determines whether the distance distance(t) input from the distance calculation unit 10 is greater (longer) than the preset transition completion threshold dist_th_out (step S518, distance(t)>dist_th_out).

[0213] If it is determined in step S518 that the distance distance(t) is not greater than the transition completion threshold dist_th_out (step S518: N), the state management unit 12 proceeds to step S518 after a certain time has elapsed (t←t+1) (step S519).

[0214] If the state management unit 12 determines in step S518 that the distance distance(t) is greater than the transition completion threshold dist_th_out (step S518: Y), the state management unit 12 proceeds to step S501 and sets the state SS of the transition to waiting S1, thereby ending the transition.

[0215] As described above, according to the video display processing device 1 of the embodiment of the present invention, the state management unit 12 manages the transition state SS (waiting S1, expanding S2, expanded S3, shrinking S4 and reduced S5) of the spherical object 3, generates a control signal according to the transition state SS and outputs it to the zoom unit 13, material change unit 15, video playback control unit 17 and other object display control unit 20.

[0216] Specifically, if the state management unit 12 determines that the distance distance(t) is not smaller than the transition start threshold dist_th_in, it determines that the user 100 is not within the sphere of the icon 101 and keeps the transition state SS at waiting S1.

[0217] In this case, the initial settings for the spherical object 3 are set to a size scaling factor scale_magni(t) = 1, a spherical image showing a still image, and unlit material, while the other objects are set with display / non-display information indicating whether they are displayed.

[0218] This displays a 3DCG model image including a thumbnail icon 101. Also, since a portion of the opening frame of the 360-degree image is pasted as a thumbnail on the inside of the icon sphere as seen from the viewpoint of the user 100, the closer the user 100 gets to the icon 101, the more the display of the icon 101 as seen by the user 100 and the display of the 3DCG model image of the outside world will geometrically match.

[0219] Furthermore, if the state management unit 12 determines that the distance distance(t) is smaller than the transition start threshold dist_th_in, it determines that the user 100 has entered the sphere of the icon 101, and changes the transition state SS from waiting S1 to expanding S2. Then, the state management unit 12 generates and outputs a control signal indicating the start of the 360-degree video, a control signal indicating the foreground material, a control signal indicating expansion, and a control signal indicating the non-display of other objects.

[0220] In this case, the spherical object 3 is set to a size that is a gradually increasing magnification ratio scale_magni(t), and is also set with a spherical image showing the played 360-degree image and a foreground material, etc., while other objects are set with displayability information indicating whether they are hidden.

[0221] As a result, the foreground material is set for the spherical object 3, so the played 360-degree video is displayed in the foreground, and the other spherical objects 3 and 3DCG objects 4 have display availability information set to indicate they are not displayed, so the video of the other spherical objects 3 and 3DCG objects 4 is not displayed. This makes it possible to compensate for the drawback of parts of the video of the 3DCG model being missing or distorted.

[0222] Furthermore, if the state management unit 12 determines that the enlargement / reduction rate scale_magni(t) is equal to or greater than the enlargement completion threshold max_scale_magni, it determines that the enlargement of the screen sphere is complete, and changes the transition state SS from Enlarging S2 to Enlarged S3. Then, the state management unit 12 generates and outputs a control signal indicating that the enlargement has stopped.

[0223] This completes the expansion of the 360-degree video being played, and just as with the transition to S2 during expansion, it is possible to compensate for the drawback of parts of the 3DCG model being missing or distorted.

[0224] Furthermore, when the state management unit 12 determines that the user 100 has operated a key and input a key control signal, it changes the transition state SS from Enlarged S3 to Shrinking S4. Then, the state management unit 12 generates and outputs a control signal indicating cueing of the 360-degree video, a control signal indicating shrinking, and a control signal indicating display of other objects.

[0225] In this case, the spherical object 3 is set to a size that is a gradually decreasing magnification ratio scale_magni(t), and the spherical image of the first frame of the played 360-degree video is set, and display availability information indicating whether or not the other objects are displayed is set.

[0226] As a result, the opening frame of the played 360-degree video of the spherical object 3 gradually becomes smaller as it is displayed, and the videos of the other objects are also displayed. Also, as in the cases of expanding S2 and expanded S3, the foreground material is set for the spherical object 3, so the played 360-degree video is displayed in the foreground and does not overlap with other objects and the 360-degree video.

[0227] Furthermore, if the state management unit 12 determines that the scaling factor scale_magni(t) is smaller than 1.0, it determines that the reduction of the screen sphere has been completed, and changes the transition state SS from "Shrinking" S4 to "Shrunken" S5. Then, the state management unit 12 generates and outputs a control signal indicating the stop of reduction, a control signal indicating the end of the 360-degree video, and a control signal indicating unlit material.

[0228] In this case, the spherical object 3 is set to a size of scale_magni(t) that is smaller than 1.0, and is also set with a spherical image showing a still image, unlit material, and the like.

[0229] As a result, the screen sphere is replaced by an icon sphere, and thumbnails are displayed. In addition, since unlit material is set for the spherical object 3, images of thumbnails and other objects are displayed according to the distance from the viewpoint of the user 100.

[0230] Furthermore, if the state management unit 12 determines that the distance distance(t) is greater than the transition completion threshold dist_th_out, it determines that the user 100 has left the sphere of the icon 101, and transitions the transition state SS from reduced S5 to waiting S1.

[0231] In this way, in the world of the 3DCG model video, a 360-degree video sphere is placed at the shooting position of the 360-degree video, and a portion of the opening frame of the 360-degree video is pasted as a thumbnail inside the icon sphere as seen from the viewpoint of user 100, so that by approaching the icon sphere, the appearance of the 6DoF world and the 3DoF world will geometrically match.

[0232] When the transition state SS is expanding S2, the entire field of view is gradually covered by the screen sphere, so there is no need to worry about the 3DCG model image and the 360-degree image overlapping, or the 3DCG model image being lost.

[0233] When the screen is expanding (S2), already expanded (S3), or shrinking (S4), the foreground material is applied to the 360-degree image, so the 3DCG model image is not displayed in front of the screen sphere, and the 3DCG model image and the 360-degree image do not overlap.

[0234] Furthermore, when enlarging S2 and enlarged S3, display availability information is set to indicate that the 3DCG model image is not to be displayed, and when reducing S4, reduced S5 and waiting S1, display availability information is set to indicate that the 3DCG model image is to be displayed, so the 3DCG model image is displayed in places other than the screen sphere during reduction and in places other than the icon sphere after reduction is complete, eliminating concerns that the 3DCG model image will not be visible.

[0235] Furthermore, as the user 100 approaches the icon 101, the appearance (thumbnail) of the interior of the icon sphere geometrically overlaps with the 3DCG model image behind it. Furthermore, when the transition starts by transitioning to Enlarging S2, a continuous video is displayed instead of the thumbnail of the 360-degree image. Furthermore, the transition state SS successively transitions from Waiting S1, Enlarging S2, Enlarged S3, Shrinking S4, and Shrinked S5, thereby achieving visual continuity. Even when transitioning to Shrinked S5, the user 100 is considered to be near the icon 101, so the user 100 can visually return from the world of the 360-degree image to the world of the 3DCG model image.

[0236] Furthermore, by having the icon 101 exist in mid-air as an unrealistic spherical object with an intuitive design, the user 100 is provided with an affordance that anticipates some kind of interaction. Furthermore, since the opening frame of the 360-degree video is used as a thumbnail for the icon, the user 100 can easily recognize the content of the video.

[0237] Therefore, by presenting a photogrammetry-based 3DCG model image and a 360-degree image in a complementary manner, it is possible to compensate for the drawback of parts of the 3DCG model image being missing, etc. Furthermore, by geometrically matching both images, it is possible to achieve visual continuity.

[0238] [Another Example / First Modification of the Transition Function Unit 2] Next, a description will be given of another example (first modified example) of the transition function section 2 shown in Fig. 3. Fig. 10 is a block diagram showing a first modified example of the configuration example of the transition function section 2.

[0239] This transition function unit 2' includes a distance calculation unit 10, a key input unit 11, a status management unit 12, a zoom unit 13', a material recording unit 14, a material change unit 15, a 360-degree video recording unit 16, a video playback control unit 17, a still image extraction unit 18, a video selection unit 19, an other object display control unit 20, an audio recording unit 21, and an audio selection unit 22.

[0240] Comparing the transition function unit 2 shown in Figure 3 with this transition function unit 2', both transition function units 2 and 2' have in common the fact that they are equipped with a distance calculation unit 10, a key input unit 11, a state management unit 12, a material recording unit 14, a material change unit 15, a 360-degree video recording unit 16, a video playback control unit 17, a still image extraction unit 18, a video selection unit 19, and an other object display control unit 20.

[0241] In contrast, the transition function unit 2' differs from the transition function unit 2 in that it includes a scaling unit 13' that is different from the scaling unit 13 included in the transition function unit 2, and further includes an audio recording unit 21 and an audio selection unit 22. In Figure 10, parts that are common to Figure 3 are given the same reference numerals as in Figure 3, and detailed explanations thereof will be omitted.

[0242] The scaling unit 13′ performs the same processing as the scaling unit 13 shown in FIG. 3, and further outputs the scaling factor scale_magni(t) as the size of the spherical object 3 to the audio selection unit 22.

[0243] The audio selection unit 22 receives the scaling factor scale_magni(t), which is the size of the spherical object 3, from the scaling unit 13'.

[0244] When the scaling rate scale_magni(t), which has been constant, changes to a value greater than the constant value, the sound selection unit 22 determines that the screen sphere has started to expand (determines that the transition state SS has transitioned from waiting S1 to expanding S2), selects the sound effect (SE: Sound Effect) at the start of expansion by reading it from the sound recording unit 21, and sets audio information indicating the sound effect to the spherical object 3.

[0245] Furthermore, when the scaling rate scale_magni(t), which had been maintained at a constant value, changes to a value smaller than the constant value, the sound selection unit 22 determines that the screen sphere has started to shrink (determines that the transition state SS has transitioned from Enlarged S3 to Shrinking S4), and selects the sound effect at the start of shrinking by reading out the sound effect at the start of shrinking from the sound recording unit 21, and sets audio information indicating that sound effect to the spherical object 3.

[0246] The sound recording section 21 records a sound effect when the screen sphere starts to expand (at the start of expansion) and a sound effect when the screen sphere starts to shrink (at the start of shrinkage).

[0247] In this case, the renderer 6 plays back the sound effect indicated by the audio information set in the spherical object 3 in accordance with the audio information.

[0248] As described above, according to the video display processing device 1 of the embodiment of the present invention, when the transition state SS transitions from standby S1 to expanding S2, the audio selection unit 22 of the transition function unit 2′ selects a sound effect indicating that the screen sphere has started to expand, and sets audio information indicating the sound effect in the spherical object 3. Furthermore, when the transition state SS transitions from expanded S3 to shrinking S4, the audio selection unit 22 selects a sound effect indicating that the screen sphere has started to shrink, and sets audio information indicating the sound effect in the spherical object 3. As a result, when the renderer 6 plays back this sound effect, the user 100 can recognize the change in the screen sphere through the sound effect.

[0249] In this way, similar to the transition function unit 2 shown in Figure 3, by presenting a photogrammetric 3DCG model image and a 360-degree image in a complementary manner, it is possible to compensate for the drawback of parts of the 3DCG model image being missing, and to geometrically match both images to achieve visual continuity. Furthermore, the awkward visual sensation of momentary distortion that occurs when the 360-degree image screen sphere is expanding and contracting can be reduced by using sound effects.

[0250] Although the audio recording unit 21 is configured to record a sound effect when the screen sphere starts to expand and a sound effect when the screen sphere starts to shrink, other sound effects (including silence) may also be recorded. For example, the audio recording unit 21 may record a different sound effect for each transition of the state SS of the transition. Furthermore, the audio recording unit 21 may record a different sound effect for each transition of the state SS of the transition. In this case, when the audio selection unit 22 determines a transition of the state SS or a state SS of the transition, the audio selection unit 22 reads out a sound effect corresponding to the transition of the state SS or the state SS from the audio recording unit 21, selects the sound effect, and sets the audio information of the sound effect to the spherical object 3. This allows the user 100 to recognize the transition of the state SS of the transition or the state SS itself through the sound effect.

[0251] [Another Example / Second Modification of the Transition Function Unit 2] Next, a description will be given of another example (second modified example) of the transition function section 2 shown in Fig. 3. Fig. 11 is a block diagram showing a second modified example of the configuration example of the transition function section 2.

[0252] This transition function unit 2" includes a distance calculation unit 10, a key input unit 11, a state management unit 12', a zoom unit 13', a material recording unit 14, a material change unit 15, a 360-degree video recording unit 16, a video playback control unit 17, a still image extraction unit 18, a video selection unit 19, an other object display control unit 20, an audio recording unit 21, an audio selection unit 22, and a movement unit 23.

[0253] Comparing the transition function unit 2′ shown in FIG. 10 with this transition function unit 2″, both transition function units 2′, 2″ have in common the fact that they are equipped with a distance calculation unit 10, a key input unit 11, a zoom unit 13′, a material recording unit 14, a material modification unit 15, a 360-degree video recording unit 16, a video playback control unit 17, a still image extraction unit 18, a video selection unit 19, an other object display control unit 20, an audio recording unit 21, and an audio selection unit 22.

[0254] In contrast, the transition function unit 2'' differs from the transition function unit 2' in that it has a state management unit 12' that is different from the state management unit 12 provided in the transition function unit 2', and further has a movement unit 23. In Figure 11, parts that are common to Figure 10 are given the same symbols as in Figure 10, and detailed explanations of them will be omitted.

[0255] 3 and 10, and further generates a control signal indicating operation when the transition state SS is standby S1, and outputs the control signal to the movement unit 23. Furthermore, when the transition state SS is other than standby S1, the state management unit 12' generates a control signal indicating stop, and outputs the control signal to the movement unit 23.

[0256] The movement unit 23 receives a control signal indicating movement or stoppage from the state management unit 12′. When the movement unit 23 receives a control signal indicating movement, it sets movement information indicating movement to the spherical object 3, and when the movement unit 23 receives a control signal indicating stoppage, it sets movement information indicating stoppage to the spherical object 3.

[0257] In this case, when rendering the spherical object 3, if the motion information set for the spherical object 3 indicates a movement, the renderer 6 generates a display image including an image of the icon ball so that the icon 101 (icon ball) of the spherical object 3 moves slowly up and down at a predetermined speed, and outputs the image to the display device 7.

[0258] In addition, when the motion information set for the spherical object 3 is "stopped," the renderer 6 generates a display image including an image of the icon ball in which the icon 101 of the spherical object 3 has stopped moving (is stopped) up and down, etc., and outputs the image to the display device 7.

[0259] As described above, according to the video display processing device 1 of the embodiment of the present invention, the movement unit 23 of the transition function unit 2" sets movement information of the action of moving the icon 101 up and down, etc. to the spherical object 3 when the transition state SS is standby S1, and sets stop movement information of stopping the movement of the icon 101 to the spherical object 3 when the transition state SS is other than standby S1. This allows the user 100 to easily recognize the position where the icon 101 is located in the virtual space when the transition state SS is standby S1.

[0260] Therefore, in addition to the same effect as the transition function unit 2' shown in Figure 10, it is possible to easily recognize where the icon 101 is located in the world of the 3DCG model image and what kind of 360-degree image is present.

[0261] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0262] For example, the transition function unit 2" of the second modified example shown in Figure 11 is equipped with an audio recording unit 21, an audio selection unit 22, and a movement unit 23, but the transition function unit 2 of the third modified example may not be equipped with the audio recording unit 21 and the audio selection unit 22, but may instead be equipped with the movement unit 23. In this case, the transition function unit 2 of the third modified example is equipped with a distance calculation unit 10, a key input unit 11, a state management unit 12', a zoom unit 13, a material recording unit 14, a material change unit 15, a 360-degree video recording unit 16, a video playback control unit 17, a still image extraction unit 18, a video selection unit 19, an other object display control unit 20, and a movement unit 23.

[0263] A typical computer can be used as the hardware configuration of the video display processing device 1 according to the embodiment of the present invention. The video display processing device 1 is configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.

[0264] The functions of the 3DCG model object, camera object 5, and renderer 6, which are comprised of transition function units 2-1, 2-2,...,2-N, spherical objects 3-1, 3-2,...,3-N, and 3DCG objects 4-1, 4-2,...,4-M, etc., provided in the video display processing device 1, are each realized by having the CPU execute a program that describes these functions.

[0265] In this case, the transition function units 2-1, 2-2, ..., 2-N may be not only the transition function unit 2 shown in Figure 3, but also the transition function unit 2' of the first modified example shown in Figure 10 or the transition function unit 2" of the second modified example shown in Figure 11.

[0266] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk), an optical disk (e.g., a CD-ROM, a DVD), or a semiconductor memory and distributed, or can be transmitted and received via a network. [Explanation of symbols]

[0267] 1. Video display processing device 2 Transition function section 3 Spherical Objects 4. 3DCG Objects 5 Camera Object 6 Renderer 7 Display device 10 Distance calculation unit 11 Key input section 12 State Management Unit 13 Zoom section 14 Material Recording Section 15 Material change section 16 360-degree video recording section 17 Video playback control unit 18 Still image extraction section 19 Video selection section 20 Other object display control section 21 Audio Recording Section 22 Audio selection section 23 Sports Club 100 users 101 Icons 102,103 objects SS status S1 Standby S2 Expanding S3 Expanded S4 Shrinking S5 Reduced distance(t) distance scale_magni(t) Magnification ratio max_scale_magni Enlargement completion threshold min_scale_magni Reduction completion threshold dist_th_in transition start threshold dist_th_out transition completion threshold

Claims

1. A video display processing device that arranges a plurality of spherical objects in a virtual space in which a plurality of 3DCG objects are arranged by photogrammetry, and displays a 3DCG model video including the 3DCG objects that are within the field of view of the user among the plurality of 3DCG objects according to a viewpoint position and a line of sight of the user in the virtual space, and also displays 360-degree video of the spherical objects that are within the field of view of the user among the plurality of spherical objects as icons, a corresponding spherical object is attached to one of the plurality of spherical objects; a plurality of transition function units that sequentially transition the transition state of the corresponding spherical object from standby S1, expanding S2, expanded S3, contracting S4, and contracted S5, and set predetermined data corresponding to the transition state to the spherical object; a plurality of spherical objects attached to one corresponding transition function unit among the plurality of transition function units, the spherical objects being set with the predetermined data by the corresponding transition function unit; a 3DCG model object including the plurality of 3DCG objects, in which predetermined data is set for each of the plurality of 3DCG objects; a camera object in which predetermined data including the viewpoint position and the line of sight direction of the user is set; a renderer that identifies the spherical object and the 3DCG object that are within the field of view of the user based on the viewpoint position and line of sight direction of the user that are set in the camera object, renders the identified spherical object, and also renders the 3DCG model object that includes the identified 3DCG object, generates a display image, and outputs the generated image to a display device, The transition function unit If the user is away from the corresponding spherical object by a distance equal to or greater than a first predetermined distance, the state of the transition is set to the standby state S1, and data for displaying a still image of the 360-degree video as a thumbnail is set to the spherical object; when the user approaches the spherical object closer than the first predetermined distance, the transition state is changed from the standby state S1 to the expanding state S2, and data for gradually expanding the 360-degree video and displaying it in the foreground is set in the spherical object; When the size of the spherical object reaches a first predetermined size, the state of the transition is changed from the expanding state S2 to the expanded state S3, and data for stopping the expansion of the 360-degree video is set in the spherical object; When there is a key input from the user or when the playback of the 360-degree video is completed, the state of the transition is changed from the enlarged state S3 to the reduced state S4, and data for gradually reducing and displaying the 360-degree video is set in the spherical object; When the size of the spherical object becomes smaller than a second predetermined size, the state of the transition is changed from the shrinking state S4 to the reduced state S5, and data for stopping the reduction of the 360-degree video and canceling the application of the foreground display is set in the spherical object; When the user moves away from the corresponding spherical object by more than a second predetermined distance, the transition state is changed from the reduced state S5 to the standby state S1.

2. 2. The video display processing device according to claim 1, The spherical object is attached to a corresponding transition function unit among the plurality of transition functions; The predetermined data is set to include a spherical object position, which defines the shooting position of the 360-degree image of the corresponding spherical object as the placement position of the spherical object, a size of the image of the spherical object, a spherical image showing the image in which the spherical object is displayed, a material that defines the rendering rules for the spherical object, and display / non-display information that indicates whether the spherical image is displayed or not; The 3DCG model object is an object including the plurality of 3DCG objects, wherein the predetermined data including display / non-display information indicating a position of the 3DCG object and whether an image of the spherical object is to be displayed or not is set for each of the plurality of 3DCG objects; The transition function unit If the user is away from the corresponding spherical object by a distance equal to or greater than a first predetermined distance, the state of the transition is set to the standby state S1, and the spherical object is set to the spherical image showing a still image of the first frame of the 360-degree image; When the user approaches the spherical object closer than the first predetermined distance, the transition state is changed from the standby S1 to the expanding S2, and the size for gradually expanding the image of the spherical object, the spherical image showing the played 360-degree image when the played 360-degree image is set as a played 360-degree image, and a foreground material indicating that the played 360-degree image is displayed in the foreground are set to the spherical object, and displayability information indicating non-display is set to the spherical objects other than the spherical object among the plurality of spherical objects and the plurality of 3DCG objects, When the size of the spherical object reaches a first predetermined size, the state of the transition is changed from the expanding state (S2) to the expanded state (S3), and the size for stopping the expansion of the image of the spherical object is set to the spherical object; When there is a key input from the user or when playback of the 360-degree video is completed, the state of the transition is changed from the enlarged state S3 to the shrinking state S4, the size for gradually shrinking the video of the spherical object is set to the spherical object, and displayability information indicating display is set to the spherical objects other than the spherical object among the plurality of spherical objects and the plurality of 3DCG objects, When the size of the spherical object becomes smaller than a second predetermined size, the state of the transition is changed from the shrinking state S4 to the reduced state S5, and the size for stopping the reduction of the image of the spherical object, the spherical image showing a still image of the first frame of the 360-degree image, and an unlit material with the foreground material removed are set to the spherical object; When the user moves away from the corresponding spherical object by more than a second predetermined distance, the transition state is changed from the contracted state (S5) to the standby state (S1); The renderer When the state of the transition is the standby S1, the still image is pasted on the inside of the spherical surface of the spherical object, and rendering is performed at the viewpoint position and the line of sight direction set in the camera object, with the outside of the spherical surface being excluded by culling and only the inside being the target, thereby generating a thumbnail of the icon sphere as seen from the user's viewpoint, and generating the display video including the thumbnail of the icon sphere; When the state of the transition is expanding S2, the reproduced 360-degree video is pasted onto the inside of the spherical surface of the spherical object, and rendering is performed at the viewpoint position and the line of sight direction set for the camera object, thereby generating a video of a screen sphere that grows over time according to the size and is displayed in the foreground in accordance with the foreground material, and for the spherical objects other than the spherical object and the plurality of 3DCG objects, rendering is not performed in accordance with the display availability information indicating non-display, and the display video is generated, which includes the video of the screen sphere, but does not include the videos of the spherical objects other than the spherical object and the videos of the plurality of 3DCG objects, and the video of the screen sphere is displayed in the foreground, When the state of the transition is the enlarged state S3, the enlargement is stopped by rendering the spherical object, and an image of the screen sphere is generated that is displayed in the foreground, and the display image is generated without including images of the spherical objects other than the spherical object and images of the plurality of 3DCG objects, and the image of the screen sphere is displayed in the foreground; When the state of the transition is shrinking S4, for the spherical object, the opening frame of the reproduced 360-degree video is pasted inside the surface of the sphere, and rendering is performed at the viewpoint position and the line of sight set for the camera object, thereby generating an image of a screen sphere that becomes smaller over time according to the size and is displayed in the foreground; for the spherical objects other than the specified spherical object and the plurality of 3DCG objects, rendering is performed in accordance with display availability information indicating the display, and the display image is generated, which includes images of the spherical objects other than the specified spherical object and images of the plurality of 3DCG objects, and in which the image of the screen sphere is displayed in the foreground; When the transition state is the reduced state S5, a thumbnail of the icon sphere is generated by rendering the spherical object, and the display image is generated including a thumbnail of the icon sphere with the foreground display application canceled.

3. 3. The video display processing device according to claim 2, The transition function unit a distance calculation unit that calculates a distance (t) between the viewpoint position of the user set in the camera object and the spherical object position set in the spherical object, using the viewpoint position of the user set in the camera object and the spherical object position set in the spherical object; a key input unit that outputs a key control signal in response to a key operation by the user; a scaling unit that calculates a scaling ratio scale_magni(t) as the size of the image of the spherical object by dividing the radius of the screen sphere by a preset radius of the icon sphere; If the distance distance(t) calculated by the distance calculation unit is smaller than a preset transition start threshold dist_th_in, the state of the transition is transitioned from the standby state S1 to the expanding state S2; When the enlargement / reduction ratio scale_magni(t) calculated by the enlargement / reduction unit is equal to or greater than a preset enlargement completion threshold max_scale_magni, the state of the transition is transitioned from the enlargement S2 to the enlarged S3; When the key control signal output from the key input unit is input, or when the playback of the 360-degree video is completed, the state of the transition is changed from the enlarged state S3 to the reduced state S4, If the scaling ratio scale_magni(t) is smaller than a preset scaling completion threshold min_scale_magni, the state of the transition is changed from S4 of the scaling in progress to S5 of the scaled-down image; a state management unit that transitions the state of the transition from the reduced state (S5) to the standby state (S1) when the distance (distance(t)) is greater than a preset transition completion threshold (dist_th_out); 1. A video display processing device comprising:

4. 4. The video display processing device according to claim 3, 10. A video display processing device, wherein the transition completion threshold dist_th_out is set to a value greater than the transition start threshold dist_th_in.

5. 5. The video display processing device according to claim 3, 2. A video display processing device, wherein the reduction completion threshold min_scale_magni is set to 1.

0.

6. 6. The video display processing device according to claim 3, The transition function unit further A material change unit, a video playback processing unit, and an other object display control unit are provided, The state management unit When the state of the transition is changed from the standby state S1 to the expanding state S2, a control signal indicating the start of the 360-degree video is output to the video playback processing unit, a control signal indicating the foreground material is output to the material change unit, a control signal indicating expansion is output to the expansion / reduction unit, and a control signal indicating the non-display is output to the other object display control unit, When the state of the transition is changed from the enlarged state S2 to the enlarged state S3, a control signal indicating stop of enlargement is output to the enlargement / reduction unit; When the state of the transition is changed from the enlarged state S3 to the reduced state S4, a control signal indicating a cue of the 360-degree video is output to the video playback processing unit, a control signal indicating reduction is output to the enlargement / reduction unit, and a control signal indicating the display is output to the other object display control unit, When the state of the transition is changed from the reducing state S4 to the reduced state S5, a control signal indicating the stop of reduction is output to the enlargement / reduction unit, a control signal indicating the end of the 360-degree video is output to the video playback processing unit, and a control signal indicating the unlit material is output to the material change unit, The scaling unit When a control signal indicating the enlargement is input from the state management unit, the size is calculated so that the image of the spherical object gradually enlarges, and the size is set to the spherical object; When a control signal indicating the stop of enlargement is input, the calculation of the size is stopped so that the enlargement of the image of the spherical object is stopped, and the size is set to the spherical object; When a control signal indicating the reduction is input, the size is calculated so that the image of the spherical object gradually reduces, and the size is set to the spherical object; When a control signal indicating the stop of the reduction is input, the calculation of the size is stopped so that the reduction of the image of the spherical object is stopped, and the size is set to the spherical object; The material change unit When a control signal indicating the foreground material is input from the state management unit, the foreground material is set to the spherical object; When a control signal indicating the unlit material is input, the unlit material is set to the spherical object; The video playback processing unit When a control signal indicating the start of the 360-degree video is input from the state management unit, the 360-degree video is played back, and the spherical video representing the played back 360-degree video is set to the spherical object; When a control signal indicating a cue to the 360-degree video is input, the 360-degree video is cue-played, and the spherical video indicating the opening frame of the played 360-degree video is set as the spherical object; When a control signal indicating the end of the 360-degree video is input, a first frame is extracted from the 360-degree video, and the spherical video showing a still image of the first frame is set as the spherical object; The other object display control unit when a control signal indicating the non-display is input from the state management unit, displayability information indicating the non-display is set to the spherical objects other than the spherical object among the plurality of spherical objects and to the plurality of 3DCG objects; A video display processing device characterized in that, when a control signal indicating the display is input from the state management unit, displayability information indicating the display is set to the spherical objects other than the spherical object among the plurality of spherical objects and to the plurality of 3DCG objects.

7. 7. The video display processing device according to claim 6, The video playback processing unit As an operation mode, one of a loop setting, a stop setting, and an end setting is set in advance, If the loop setting is preset, when the playback of the 360-degree video is completed, the playback continues from the first frame of the 360-degree video; If the stop setting is set in advance, when the playback of the 360-degree video is completed, the playback is paused at the final frame of the 360-degree video, If the end setting is preset, when the playback of the 360-degree video is completed, a playback status indicating that playback is completed is output to the status management unit; The state management unit 2. A video display processing device, comprising: a video display processing unit that sets the state of the transition to the reducing state S4 when the playback state of the playback completion is input from the video playback processing unit;

8. 3. The video display processing device according to claim 2, The transition function unit Furthermore, sound information indicating a sound effect corresponding to the state of the transition or the transition of the state is set to the spherical object; The renderer a spherical object that is configured to receive the sound from the spherical object;

9. 9. The video display processing device according to claim 2 or 8, The transition function unit Furthermore, when the state of the transition is the standby S1, motion information indicating a motion is set to the spherical object; The renderer 2. A video display processing device comprising: a display of the icon sphere moving up and down at a predetermined speed when the motion information set for the spherical object indicates the motion.

10. a computer that constitutes an image display processing device that arranges a plurality of spherical objects in a virtual space in which a plurality of 3DCG objects are arranged by photogrammetry, and displays a 3DCG model image including the 3DCG object that is within the field of view of the user among the plurality of 3DCG objects according to the viewpoint position and line of sight of the user in the virtual space, and also displays a 360-degree image of the spherical object that is within the field of view of the user among the plurality of spherical objects as an icon; a corresponding spherical object is attached to one of the plurality of spherical objects; a plurality of transition function units in a case where the transition function units sequentially transition the transition state of the corresponding spherical object from standby S1, expanding S2, expanded S3, shrinking S4, and reduced S5, and set predetermined data corresponding to the transition state to the spherical object; a plurality of spherical objects in a case where the plurality of spherical objects are attached to one corresponding transition function unit among the plurality of transition function units, and the predetermined data is set by the corresponding transition function unit; a 3DCG model object including the plurality of 3DCG objects, in which predetermined data is set for each of the plurality of 3DCG objects; a camera object to which predetermined data including the viewpoint position and the line of sight direction of the user is set; and a program for functioning as a renderer that identifies the spherical object and the 3DCG object that are within the field of view of the user based on the viewpoint position and line of sight of the user that are set in the camera object, renders the identified spherical object, and also renders the 3DCG model object that includes the identified 3DCG object, generates a display image, and outputs the generated image to a display device, The transition function unit If the user is away from the corresponding spherical object by a first predetermined distance or more, the state of the transition is set to the standby state S1, and data for displaying a still image of the 360-degree video as a thumbnail is set to the spherical object; when the user approaches the spherical object closer than the first predetermined distance, the transition state is changed from the standby state S1 to the expanding state S2, and data for gradually expanding the 360-degree video and displaying it in the foreground is set in the spherical object; When the size of the spherical object reaches a first predetermined size, the state of the transition is changed from the expanding state S2 to the expanded state S3, and data for stopping the expansion of the 360-degree video is set in the spherical object; When there is a key input from the user or when the playback of the 360-degree video is completed, the state of the transition is changed from the enlarged state S3 to the reduced state S4, and data for gradually reducing and displaying the 360-degree video is set in the spherical object; When the size of the spherical object becomes smaller than a second predetermined size, the state of the transition is changed from the shrinking state S4 to the reduced state S5, and data for stopping the reduction of the 360-degree video and canceling the application of the foreground display is set in the spherical object; When the user moves away from the corresponding spherical object by more than a second predetermined distance, the state of the transition is changed from the reduced state S5 to the waiting state S1.

Images