VR motion picture providing system, VR motion picture proving method, and program

Abstract

To provide an arrangement for displaying information on a VR image.SOLUTION: A VR motion picture providing system according to the present invention has a VR motion picture outputting unit for outputting a virtual reality (VR) 360 degree motion picture, a reception unit for receiving an input relating to the VR 360 degrees motion picture, and an information outputting unit for outputting predetermined information when the reception unit receives the input.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to a VR video providing system, a VR video providing method, and a program. [Background technology]

[0002] As background art in this technical field, there is JP2019-071040A (Patent Document 1). The publication states that "the advertising system 100 includes a cloud server 200 which includes a communication unit 210 capable of communicating with a first terminal 300 and a second terminal 400, a cultural property information unit 220 which displays cultural property information including 360° VR images on the second display unit 430 of the second terminal, an advertising information unit 230 which displays advertising information on the second display unit 430 of the second terminal, a counting unit 260 which counts the number of accesses from the second terminal 400 to the cloud server 200, and a cultural property protection fee calculation unit 270 which calculates an amount to be paid by the user of the first terminal 300 as a cultural property protection fee to shrines and temples that possess cultural properties, based on the number of accesses counted by the counting unit 260. The advertising information is displayed on the VR images, which are cultural property information, displayed on the second display unit 430 of the second terminal via the communication unit 210" (see abstract). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-071040 A Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned Patent Document 1 describes a technique for displaying information such as advertisement information on a VR image, which is cultural property information displayed on a display unit of a second terminal via a communication unit. However, the technique of Patent Document 1 does not consider a mechanism for displaying information on a VR image. Therefore, the present invention provides a mechanism for displaying information on a VR image. [Means for solving the problem]

[0005] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problems, and one example is characterized by providing a VR video providing system including a VR video output unit that outputs 360-degree Virtual Reality (VR) video, a reception unit that receives input related to the VR 360-degree video, and an information output unit that outputs specified information when the reception unit receives the input. Effect of the Invention

[0006] According to the present invention, a mechanism for displaying information on a VR image can be provided. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is an example of an overall configuration diagram of a VR video providing system 100. [Diagram 2] FIG. 2 shows an example of the hardware configuration of the management server 101. [Diagram 3] FIG. 3 shows an example of the hardware configuration of the user terminal 102. [Figure 4] FIG. 4 is an example of user information 400. [Diagram 5] FIG. 5 is an example of VR 360-degree video information 500. [Figure 6] FIG. 6 is an example of the EC site information 600. [Figure 7] FIG. 7 is an example of a user registration flow 700. [Figure 8] FIG. 8 shows an example of a VR 360-degree video reception flow 800. [Figure 9]FIG. 9 is an example of a VR 360-degree video playback flow 900. [Figure 10] FIG. 10 is an example of a site access flow 1000. [Figure 11] FIG. 11 is an example of an EC sales flow 1100. [Figure 12] FIG. 12 is another example of an EC sales flow 1200. [Figure 13] FIG. 13 is an example of a conceptual diagram 1300 of a VR 360-degree video. [Figure 14] FIG. 14 is another example of a conceptual diagram 1400 of a VR 360-degree video. [Figure 15] FIG. 15 is an example of a visual image 1500 of VR 360-degree video content. [Figure 16] FIG. 16 is another example of a visual image 1600 of VR 360-degree video content. [Figure 17] FIG. 17 is another example of a visual image 1700 of a VR 360-degree video. [Figure 18] FIG. 18 is another example of a conceptual diagram 1800 of a VR 360-degree video. [Figure 19] FIG. 19 is an example of a display image 1900 on an EC site. [Figure 20] FIG. 20 shows another example of a display image 2000 on an EC site. [Figure 21] 21(A) to (C) are examples of visual images 2100 of VR 360-degree video content. [Figure 22] 22(A) to (C) are other examples of visual images 2200 of VR 360-degree video content. [Diagram 23] FIG. 23 is another example of a visual image 2300 of a VR 360-degree video. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0008] Hereinafter, an embodiment will be described with reference to the drawings. A VR video providing system 100 according to an embodiment displays predetermined information on a VR 360-degree video. Here, the VR 360-degree video is also called a celestial sphere video or a panoramic video, and refers to a video output in a seamless sphere so as to surround the viewpoint in all directions. In the following embodiment, for simplicity, a so-called 3DoF (Degrees of Freedom) type video will be described as a VR 360-degree video in which the viewpoint position is fixed and interactivity is only in the line of sight (i.e., three degrees of freedom that can rotate around each of the XYZ Cartesian coordinate system).

[0009] The VR 360-degree video in this embodiment is, for example, a virtual tour video of an arbitrary city, which introduces the lifestyles of the people in the surrounding area and activities related to the area while touring the city's famous places, historical sites, historical buildings, etc., and conveys the charm of the city. Since the virtual tour video is a VR 360-degree video, the viewer (typically a user) can get the feeling as if he or she actually went to the city, toured the city, and actually experienced the activities in the video. The VR 360-degree video is configured to have a story that gives the viewer the feeling of actually traveling in the city, for example.

[0010] FIG. 1 is an example of an overall configuration diagram of a VR video providing system 100 that provides such VR 360-degree videos. The VR video providing system 100 includes one or more management servers 101 and one or more user terminals 102, which can be connected to each other via a network. The management server 101 and the user terminal 102 can transmit and receive information to each other via the network, regardless of whether the network is wired or wireless. The user terminal 102 is a terminal used by a user who watches VR 360-degree videos. The user terminal 102 may be provided separately from the management server 101, or may be provided integrally with the management server 101.

[0011] Each terminal (including the management server) of the VR video providing system 100 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or may be a wearable terminal such as a head-mounted (HM) type, glasses type, or clothing type. These may also be stationary or portable computers, or servers located on the cloud or network. In terms of function, they may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Or, they may be a combination of a plurality of these terminals. For example, a combination of one smartphone and one wearable terminal may function logically as one terminal. They may also be other information processing terminals.

[0012] Each terminal and management server 101 of the VR video providing system 100 includes a processor that executes an operating system, an application, a program, etc., a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as an IC card, a hard disk drive, an SSD (Solid State Drive), a flash memory, etc., a communication control unit such as a network card, a wireless communication module, a mobile communication module, etc., an input device such as a touch panel, a keyboard, a mouse, a voice input device, a motion controller, an input device that detects motion by capturing an image of a camera unit, a GPS, a gyro sensor, an acceleration sensor, etc., and an output device such as a monitor, a display, a printer, a voice output device, an oscillator, etc. The output device may be a device or terminal that transmits information to be output to an external monitor, a display, a printer, a device, etc.

[0013] The main memory stores various programs and applications (software modules), and the processor executes these programs and applications to realize each functional element of the overall system. Each module may be an independent program or application, or may be implemented as a subprogram or function within an integrated program or application. Each module may be implemented as hardware (hardware module) by integrating circuits or employing a microcomputer. Furthermore, each module may be provided in a single terminal (including a management server), or may be provided separately in two or more terminals (including a management server) connected to each other via a network.

[0014] In this specification, each module is described as an entity (subject) that performs processing, but in reality, a processor that processes various programs, applications, etc. (modules) executes the processing. The auxiliary storage device stores various databases (DB). A "database" is a set of data that is organized and collected so that it can handle any data operation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The auxiliary storage device is a functional element (storage unit) that stores one or more sets of data. There are no limitations on the method of implementing the database, and it may be, for example, a database management system, spreadsheet software, or a text file such as XML or JSON.

[0015] FIG. 2 shows an example of the hardware configuration of the management server 101. The management server 101 is an element that manages the VR video providing system 100 of this embodiment, and is configured, for example, by a server placed on a cloud. The main memory device 201 stores programs and applications such as a management module 211, an execution module 212, and an EC (Electronic Commerce) sales module 213, and each functional element of the management server 101 is realized by the processor 203 executing these programs and applications.

[0016] The management module 211 manages the connection and operation of the user terminal 102. For example, the management module 211 cooperates with a user management module 311 of the user terminal 102 to control basic operations for viewing VR 360-degree video executed in the user terminal 102 using the VR video providing system 100.

[0017] The execution module 212 registers and manages user information, VR 360-degree video information, etc. The execution module 212 also cooperates with the VR video output module 312, information output module 314, event detection module 316, etc. of the user terminal 102 to display the VR 360-degree video information on the display (an example of the output device 305) of the user terminal 102.

[0018] The EC sales module 213 registers and manages EC site information, etc. Also, the EC sales module 213 cooperates with a site access module 315 of the user terminal 102 and displays information related to these EC sites on a display (an example of the output device 305) of the user terminal 102.

[0019] The auxiliary storage device 202 stores user information 400, VR 360-degree video information 500, EC site information 600, etc. These pieces of information will be described later.

[0020] FIG. 3 shows an example of the hardware configuration of the user terminal 102. The user terminal 102 in this embodiment is a stand-alone head-mounted device. However, the user terminal 102 may be, for example, a PC-connected head-mounted device, a smartphone, a tablet, a notebook PC, a desktop PC, or other terminal. The main memory device 301 stores programs and applications such as a user management module 311, a VR video output module 312, an input reception module 313, an information output module 314, a site access module 315, and an event detection module 316. The processor 303 executes these programs and applications to realize each functional element of the user terminal 102.

[0021] The user management module 311 controls the basic operation of the user terminal 102. For example, the user management module 311 cooperates with the management module 211 of the management server 101 to control the basic operation for managing the VR 360-degree video delivered in the VR video providing system 100.

[0022] The VR video output module 312 outputs VR 360-degree video to a display (an example of an output device 305) of the user terminal 102. The input receiving module 313 receives input related to VR 360-degree video. When the input receiving module 313 receives an input, the information output module 314 outputs predetermined information to the display.

[0023] The site access module 315 is an additional component that connects to a destination internet site when the input receiving module 313 receives the selection of the guide link information. The guide link information is, for example, information including a URL for connecting to the predetermined internet site. The event detection module 316 is an optional component that detects that a specific event has occurred in the VR 360-degree video output by the VR video output module 312.

[0024] 4 to 6 are examples of various information stored in the management server 101. It is assumed that some or all of this information is stored in a JSON format file, but is not limited to this. Some or all of this information may be configured to be stored in a relational database or a non-relational database.

[0025] FIG. 4 is an example of user information 400. The user information 400 stores information about users to be connected and managed by the management module 211, and typically stores information about the user of the user terminal 102, etc. The user information 400 includes, for example, a user ID, a user display ID, a user name, an address, payment information, password information, and identity verification information, and values ​​such as those shown as sample values ​​420 are input for each field name 410.

[0026] FIG. 5 is an example of VR 360-degree video information 500. The VR 360-degree video information 500 records information related to the VR 360-degree video content provided by the VR video providing system 100. The VR 360-degree video content is composed of the VR 360-degree video itself (hereinafter sometimes referred to as the "VR 360-degree video body") and specific information to be displayed superimposed on this VR 360-degree video (hereinafter sometimes referred to as the "insertion information"). The VR 360-degree video information 500 includes information related to the VR 360-degree video body and information related to the insertion information.

[0027] Of the VR 360-degree video information 500, information relating to the VR 360-degree video itself includes, for example, a VR 360-degree video ID, a VR 360-degree video name, and a VR 360-degree video storage URL. Among the VR 360-degree video information 500, the information related to the insertion information includes, for example, an insertion information ID, an insertion information name, a storage URL of the insertion information, and insertion condition information. The insertion condition is, for example, information related to the insertion time information, the insertion position, the insertion angle, the insertion form (shape) of the insertion information for the VR 360-degree video. The insertion information may be any one of these, or a combination of two or more. For example, the VR 360-degree video information 500 has values ​​such as those exemplified by sample values ​​520 input into field names 510.

[0028] FIG. 6 is an example of the EC site information 600. The EC site information 600 stores information about a destination site to which the management module 211 directs the user terminal 102 . The EC site information 600 includes, for example, an EC site ID, an EC site name, an EC site URL, and the like, and values ​​such as those shown as sample values ​​620 are input for each field name 610 .

[0029] FIG. 7 is an example of a user registration flow 700. When watching a VR 360-degree video using the VR video providing system 100, for example, the user accesses the management server 101 via the user terminal 102 (step 710). If it is the first time, the management module 211 of the management server 101 works in cooperation with the user management module 311 of the user terminal 102 to display a user registration screen on the display of the user terminal 102 (step 720) (hereinafter, displaying a screen or the like on the display of the user terminal 102 in this manner may simply be referred to as "displaying on the user terminal 102").

[0030] The management module 211 displays input fields for information required for user registration on the user registration screen. Information entered into the input fields by user operation is transmitted to the management server 101 by the user management module 311 (step 730). The management module 211 receives the sent information and stores it in the user information 400. This completes the user registration. The management module 211 transmits a notification of the completion of registration to the user terminal 102 (step 750).

[0031] FIG. 8 shows an example of a VR 360-degree video reception flow 800. Next, the user management module 311 of the user terminal 102 starts up application software for viewing VR 360-degree video, for example, by a user operation as necessary (step 810). Using this application software, the user management module 311 transmits necessary authentication information such as a user ID and a password to the management server 101 (step 820). The management module 211 of the management server 101 receives the authentication information and performs a user authentication process (step 830).

[0032] If the received authentication information matches the user information 400, the management module 211 displays, for example, a selection screen for VR 360-degree videos on the user terminal 102 (step 840). The management module 211 displays, for example, on a selection screen, information relating to one or more viewable VR 360-degree videos (for example, the name and thumbnail of the VR 360-degree video).

[0033] When a VR 360-degree video is selected by the user on the selection screen, the user management module 311 transmits information indicating that the VR 360-degree video has been selected to the management server 101 (step 850). The management module 211 accesses the VR 360-degree video information 500 in the auxiliary storage device 202 (step 860) and transmits the VR 360-degree video content (i.e., a set of the VR 360-degree video itself and insertion information) corresponding to the selected VR 360-degree video to the user terminal 102 (step 870). The user management module 311 receives the VR 360-degree video content and stores it in the auxiliary storage device 302.

[0034] FIG. 9 is an example of a VR 360-degree video playback flow 900. When a playback operation of a VR 360-degree video is performed on the user terminal 102, the VR video output module 312 accepts the playback operation of the VR 360-degree video (step 910) and displays (plays) the VR 360-degree video on the HM display (step 920). Then, when the input receiving module 313 receives input related to VR 360-degree video, or when the event detection module 316 detects a specific event (Yes in step 930), the information output module 314 outputs specific information (step 940). The input reception module 313 receives the input (step 930) and the information output module 314 outputs the information (step 940) each time (No in step 950) until the VR 360-degree video ends (Yes in step 950).

[0035] The input at step 930 is received, for example, When a specific area specified by coordinates, etc. is selected by the user through a click, etc. In this case, which area to click may be displayed so that the user can understand, for example, by an icon saying "Show explanation." Also, although coordinates or the like are specified in a hidden state so that it is not explicitly clear where to click, a configuration may be adopted in which information can be output by accepting a click. When input is received by moving the mouse or pointer into a specific area specified by coordinates, etc. (mouse over) Possible reasons include:

[0036] The detection of the event in step 930 may be performed, for example, by When the playback time of the VR360-degree video reaches a certain time In this case, the timing of inserting the information is defined in the insertion time information of the VR 360-degree video information 500. When the coordinates in the VR 360-degree video reach the specified coordinate information position In this case, the insertion position of the information is defined in the insertion position information of the VR 360-degree video information 500. Possible reasons include:

[0037] Here, we will explain VR 360-degree video content. FIG. 13 is an example of a conceptual diagram 1300 of a VR 360-degree video. 13 is composed of, for example, a collection of equirectangular panoramic images 1320 (equirectangular projection) with a horizontal:vertical ratio of 2:1, prepared at a predetermined frame rate (number of images per second), and audio data. The panoramic images 1320 are arranged in a cylindrical shape along the equator of the celestial sphere 1330, and are converted so as to be pasted onto the surface of the celestial sphere 1330.

[0038] Then, the center of the celestial sphere 1330 is surrounded by the panoramic image 1320 that has been converted into a celestial sphere, and by moving the line of sight from the center of the celestial sphere 1330 as the viewpoint 1310, the panoramic image 1320 can be viewed in all directions, 360 degrees horizontally and 180 degrees vertically. In this way, a viewer of a VR 360-degree video can view the panoramic image 1320 at any viewing angle (Roll, Pitch, Yaw) = (α, β, γ) around the X-axis, Y-axis, and Z-axis, for example, and can get the feeling that they are immersed in the world of the VR 360-degree video.

[0039] Typically, the VR 360-degree video body is a VR 360-degree image (which may be a frame) that is prepared by seamlessly stitching together (stitching) multiple omnidirectional images captured by an omnidirectional camera at a predetermined frame rate. Although not limited to this, the omnidirectional camera may be, for example, a single-lens or twin-lens camera equipped with one or two fisheye lenses, or a multi-lens camera in which multiple lenses are arranged circumferentially facing outward in the radial direction.

[0040] As the VR 360-degree image, various recording formats such as the equirectangular format, the cube map format, the mesh format (dome master format), etc. Typically, the omnidirectional camera is provided with stitching software suitable for stitching the captured omnidirectional images, and for example, stitching can be performed automatically or manually by using this stitching software.

[0041] FIG. 15 is an example of a visual image 1500 that shows how VR 360-degree video content appears. This visual image 1500 is an image (e.g., a natural view) that is visually recognized when a region of a field of view 1510 of a panoramic image 1320 (which may be one frame) for a scene in VR 360-degree video content is output to an HM display, etc. For convenience, the visual image 1500 specifically illustrates only an image of the central part (front) of the field of view 1510. 15, a visual image 1500 of this scene includes a VR 360-degree image 1520 and an explanatory image 1540 displayed in front of and superimposed on the VR 360-degree image. The explanatory image 1540 is an example of insertion information in the present technology.

[0042] The VR 360-degree image 1520 includes as its subjects, for example, a historical building (e.g., a shrine) and a signboard 1530. The signboard 1530 is a vertically elongated flat plate that is supported by leaning it up against a tree in the shrine grounds. The surface of the signboard 1530 is approximately flat. The explanatory image 1540, for example, shows explanatory information about this historical building, and provides information about the subject (an example of a target object) in the VR 360-degree image 1520.

[0043] The explanatory image 1540 is a sheet-like object superimposed on the VR 360-degree image 1520. The explanatory image 1540 has dimensions slightly smaller than the board surface of the signboard 1530, and is superimposed on the VR 360-degree image 1520 in a position and orientation as if it were attached to the signboard 1530. This explanatory image 1540 is designed to be displayed superimposed on the signboard 1530 when, for example, an event occurs in which the signboard 1530 appears in the VR 360-degree video body.

[0044] More specifically, for example, when the input receiving module 313 receives an input related to the event that a signboard 1530 appears, the information output module 314 may be configured to output an explanatory image 1540. In this case, for example, the explanatory image 1540 is mapped in advance to a location (position) corresponding to the surface of the signboard 1530 in the VR space of the VR 360-degree video body. Then, when at least a part of the location is output to the HM display as a viewing area with the distribution of the VR 360-degree video, the input receiving module 313 detects the location. Then, the information output module 314 outputs a part or all of the explanatory image 1540 corresponding to the detected output location. However, the method of outputting the explanatory image 1540 is not limited to this.

[0045] This explanatory image 1540 can be understood to be disposed within the celestial sphere 1330, for example, like the image 1340 in the conceptual diagram 1300 of Fig. 13. The dimensions of the image 1340 are exaggerated for ease of illustration. The explanation image 1540 is arranged, for example, at a predetermined position coordinate (x1, y1, z1) in a three-dimensional space based on the X-axis, Y-axis, and Z-axis of the celestial sphere 1330, in a predetermined orientation (for example, in an orientation where the normal vector is (α1, β1, γ1)).

[0046] In other words, in a certain scene, the explanatory image 1540 is mapped at a predetermined position and in a predetermined orientation within the celestial sphere 1330. For mapping, for example, a grid component or the like capable of forming a grid (mesh) at a predetermined position and in a predetermined orientation on a VR 360-degree image may be used, and the explanatory image 1540 may be fitted into the formed grid.

[0047] When the direction of the line of sight from viewpoint 1310 rotates around the X-axis, Y-axis, and / or Z-axis of celestial sphere 1330, image 1340 rotates relatively in the opposite direction around the X-axis, Y-axis, and / or Z-axis. Furthermore, when the line of sight is directed upward or downward with respect to the X-axis, Y-axis, and / or Z-axis, the image 1340 is directed downward or upward in the opposite direction relative to the X-axis, Y-axis, and / or Z-axis. The image 1340 aligned with the viewpoint position and line of sight direction of the user terminal 102 can be generated, for example, by software called a renderer.

[0048] 16 is another example of a visual image 1600 of a VR 360-degree video. In the figure, only a bulletin board 1630 and a portion of an explanatory image 1640 are specifically shown. For example, specifically, the line of sight of a user viewing visual image 1500 in Fig. 15 from the front rotates to the right. Then, as shown in Fig. 16, bulletin board 1630 and explanatory image 1640 move to, for example, the left edge portion of field of view 1610. In this case, no distortion is seen for the object in front of the field of view 1610, but the bulletin board 1630 and explanatory image 1640 in the left edge portion of the field of view 1610 appear slightly distorted. However, the position and orientation of the explanatory image 1640 in the celestial sphere 1330 in front of it have not changed.

[0049] In addition, the insertion of an explanatory image 1540 into the VR 360-degree image 1520 of each scene can be achieved by specifying the position, orientation, and size of the explanatory image 1540 to be inserted into the VR 360-degree image according to time, as shown in the example below. Although not limited to this, for example, it is possible to perform image recognition on a target part (here, a signboard 1530) whose surface displays information. For example, the surface pattern of the signboard 1530 is detected, and an explanatory image 1540 is mapped at a predetermined position on this surface.

[0050] In this way, the posture and parts of the explanatory image 1540 superimposed on the surface of the signboard 1530 can be easily changed depending on the posture and appearance of the signboard 1530 in each scene of the VR 360-degree image 1520. Such processing can be accomplished using, for example, pattern recognition software, motion graphics editing software, filter creation software, and the like.

[0051] In addition, for example, the event detection module 316 may be configured to detect whether or not at least a portion of the location corresponding to the surface of the signboard 1530 is output as a viewing area on the HM display. For example, the event detection module 316 detects the occurrence of an event "a signboard 1530 appears" in the VR 360-degree video output by the VR video output module 312. The input receiving module 313 may be configured to receive, for example, the detection of the occurrence of this event by the event detection module 316 as an input.

[0052] In this example, the information output module 314 outputs predetermined information superimposed on a signboard 1530 (an example of an object) existing in the VR space formed in the VR 360-degree video. The predetermined information is an explanation of a shrine (an example of an object) existing in the VR space formed in the VR 360-degree video. With this configuration, information prepared separately about a shrine, which is one object, can be presented as if it were actually written on another object. This has the advantage that it is easy to present specific information in the VR 360-degree video space without the user feeling uncomfortable.

[0053] Also, if necessary, only the explanatory image 1540 (information) can be replaced with another explanatory image, and more appropriate information can be presented depending on, for example, the time of year or the attributes of the user. As an example, the explanatory image 1540 including the information "Sacred talisman offering place" can be easily replaced with an explanatory image including other information such as "Shichi-Go-San," "XX family wedding," and "This month's festivals and events."

[0054] In addition, the information output module 314 outputs predetermined information to the surface of a signboard 1530 (an example of an object) that exists in the VR space formed in the VR 360-degree video. With this configuration, it is possible to display text information in a manner that blends in with the space of the VR 360-degree video. As a result, in cases where the VR 360-degree video has a storyline, for example, it is possible to present specified information without compromising the sense of worldliness of the VR 360-degree space.

[0055] The surface of the signboard 1530 is flat, and the information output module 314 outputs predetermined information onto the flat surface in the VR space formed in the VR 360-degree video. This configuration has the advantage that, when the predetermined information includes text information, a user looking directly at the signboard 1530 can easily visually recognize the information.

[0056] (Modification) FIG. 14 is another example of a conceptual diagram 1400 of a VR 360-degree video. For example, in the above embodiment, the VR 360-degree video was in 3DoF format. However, the VR 360-degree video is not limited to the 3DoF format. For example, this VR 360-degree video is a so-called 6DoF format that can move (translate) along each of the X, Y, and Z axes in addition to the above 3DoF. The VR video providing system 100 can output images related to predetermined information together with the VR 360-degree image for videos in the 6DoF format.

[0057] In the case of 6DoF video, the user's viewpoint is at the center 1410 of the celestial sphere 1420 at a first time point, but can be translated in any direction in the VR space. In other words, the user's viewpoint can move from position (0,0,0) in the VR 360-degree space to position (x2,v2,z2).

[0058] In this case, the VR video output module 312 can convert a VR 360-degree image of a predetermined field of view with the viewpoint at position (0,0,0) into a VR 360-degree image with the viewpoint at position (x2,v2,z2) and output the VR 360-degree image. Also, when the input receiving module 313 receives input related to a VR 360-degree video, the information output module 314 can convert an image related to predetermined information with the viewpoint at position (0,0,0) into a VR 360-degree image with the viewpoint at position (x2,v2,z2) and output the VR 360-degree image.

[0059] In addition, if a panoramic image corresponding to the entire celestial sphere centered on the position (x2, v2, z2) of the viewpoint after the movement is prepared, the VR video output module 312 can output a VR 360-degree image with the viewpoint at the position (x2, v2, z2) based on the panoramic image. In addition, the information output module 314 can output an image relating to predetermined information that is mapped to a predetermined posture at a predetermined position in a VR 360-degree image with the position (x2, v2, z2) as the viewpoint. EXAMPLES

[0060] In the VR video providing system 100 of the second embodiment, the information output module 314 is different from that of the first embodiment in that it outputs predetermined information to a curved surface in the VR space formed in the VR 360-degree video. The other configurations of the system of the second embodiment may be the same as those of the first embodiment, and the same configurations and effects will not be described again.

[0061] FIG. 17 is another example of a visual image 1700 of a VR 360-degree video. This visual image 1700 is an image (e.g., a natural view) that is visually recognized when a field of view 1710 of a panoramic image (which may be one frame) of a scene in VR 360-degree video content is output to an HM display or the like. This visual image 1700 includes a VR 360-degree image 1720, and video information 1740 and an icon 1750 that are superimposed and displayed in front of the VR 360-degree image. The video information 1740 and the icon 1750 are an example of insertion information in the present technology.

[0062] The VR 360-degree image 1720 includes, for example, a cylindrical building 1730 as one of the subjects. The building 1730 is a vertically elongated cylindrical shape, and its surface form is a smoothly curved surface macroscopically. The video information 1740 is, for example, a video showing information about the stores in the building 1730, such as an advertising video showing sales information. The video information 1740 is, for example, a sheet-like object that is overlaid on the VR 360-degree image 1720. The video information 1740 is configured to be overlaid on the VR 360-degree image 1720 in a position and orientation that makes it appear as if it were attached to the front of the building 1730.

[0063] For example, a predetermined portion of the surface of building 1730 is set as the display area (output area) of video information 1740, and video information 1740 is mapped to this area. For example, an overlay grid is formed in a predetermined shape (shape of the display area) so as to fit the surface (curved surface) of the display area. Then, the video content corresponding to video information 1740 may be fitted into this grid.

[0064] Here, the input receiving module 313 may be configured such that, when input is received regarding an event in which a building 1730 appears in the VR 360-degree image 1720, the information output module 314 outputs (plays) video information 1740. According to such a configuration, video information 1740 is output in response to the appearance of building 1730 in user's field of view 1710. Therefore, there is an advantage that the video information 1740 can be presented to the user from the beginning, which is likely to attract the user's interest.

[0065] Furthermore, according to the above configuration, the VR video output module 312 can output video information 1740 (an example of predetermined information) to the curved surface of a three-dimensional object 1850 in the VR space formed within the VR 360-degree video. Furthermore, a user who views the video information 1740 can intuitively understand that the predetermined information included in the video information 1740 is information about a building 1730 (an example of a target object) that exists in the VR space.

[0066] Furthermore, even if the viewpoint moves, the video information 1740 moves relative to the viewpoint together with the building 1730, so that a user viewing the video information 1740 is prevented from feeling uncomfortable in the VR space. As a result, in cases where the VR 360-degree video has a story or is a video about a real-life environment, for example, it is possible to present the specified information without compromising the sense of the world in the VR 360-degree space.

[0067] FIG. 10 is an example of a site access flow 1000. The VR video output module 312 outputs, for example, an icon 1750, which is information indicating that transition to another site is possible, to the lower right of the video information 1740 (an example of the periphery of the object) together with the video information 1740 (step 1010). The information indicating that transition to another site is possible may be displayed by displaying the icon, or may be displayed by highlighting the video advertisement 1740 (an example of video information) by, for example, making the video advertisement 1740 (an example of video information) glow, changing the color of the surrounding border, blinking, or the like.

[0068] The icon 1750 is, for example, a picture of a human index finger pointing up, and indicates that the icon 1750 includes guide link information that can guide the user to an Internet site related to the store. The video advertisement 1740 is configured to be selectable, and can be selected by the user operating a joystick (an example of an input device) or a mouse, for example. The icon 1750 itself may be a guide link.

[0069] When the user selects the video advertisement 1740 via the input device, the input receiving module 313 receives the selection of the guide link information such as the video advertisement 1740 or the icon 1750 (Yes in step 1020). When the input receiving module 313 receives the selection of the guide link information, the site access module 315 connects to the guide destination Internet site (step 1030). The site access module 315 then displays the redirection site on the HM display. In this embodiment, the redirection site is an EC site where the EC sales module 213 of the management server 101 manages sales procedures, and sells products handled in stores in the building 1730.

[0070] FIG. 19 is an example of a display image 1900 on an EC site. The site access module 315 can display a screen 1920 of the redirected site within a window 1910 displaying the VR 360-degree video, for example, as shown in FIG. According to such a configuration, it is not necessary to change the output area of ​​the VR 360-degree video to the HM display, and the screen 1920 of the guided site can be displayed, and the change in the image seen by the user can be suppressed to a minimum. This is also advantageous in that it can suppress eye fatigue of the user, for example.

[0071] FIG. 20 shows another example of a display image 2000 on an EC site. The site access module 315 can display the guided site in a new window 2020 different from the window 2010 displaying the VR 360-degree video, for example, as shown in FIG. 20 . According to this configuration, the new window 2020 can be displayed in a relatively large area relative to the window 2010 displaying the VR 360-degree video. As a result, the guided site can be displayed with good visibility.

[0072] When the site access module 315 displays the guided site on the HM display, the VR video output module 312 may stop the progress of the VR 360-degree video (for example, pause the video). Alternatively, for example, the site access module 315 may display the destination site in the entire field of view of the HM display, and the VR video output module 312 may not output VR 360-degree video to the HM display. On the other hand, the site access module 315 may be configured to display screen 1920 or screen 2020 while the VR video output module 312 continues playing the VR 360-degree video.

[0073] FIG. 11 shows an example of a redirection site EC sales flow 1100. When the user terminal 102 is connected to an EC server displaying an EC site by the site access module 315, for example, the EC sales module 213 of the management server 101 displays product information on the HM display of the user terminal 102 (step 1105). Next, when an operation for selecting an item and checking stock is accepted from the user terminal 102 (step 1110), the EC sales module 213 performs an operation for checking stock of the item, for example, on the EC server (step 1115). Furthermore, the EC sales module 213 performs stock confirmation processing and delivery date confirmation processing with the EC server (step 1120), and outputs the stock information and delivery date information obtained as a result to the user terminal 102 (step 1125).

[0074] Thereafter, based on the output contents, for example, when an order confirmation and payment operation is received from the user terminal 102 (step 1130), the EC sales module 213 performs a payment process (step 1135) and places an order for the product with the EC server (step 1140). When the EC sales module 213 receives the order confirmation from the EC server (step 1145), it outputs the order confirmation information to the user terminal 102 (step 1150). After that, when the product is delivered to the management entity of the management server 101, the management entity ships the product to the user. In this way, the product can be sold by using the VR video providing system 100.

[0075] (Variation 2) FIG. 12 is another example of an EC sales flow 1200. When the user terminal 102 connects to the EC server displaying the EC site via the site access module 315, for example, the EC sales module 213 of the management server 101 outputs product information and stock information to the HM display of the user terminal 102 (step 1210).

[0076] Thereafter, based on the output contents, for example, when the EC sales module 213 receives order confirmation information and payment information for an in-stock item from the user terminal 102 (step 1220), it performs a settlement process (step 1230). Then, the EC sales module 213 outputs order confirmation information to the user terminal 102 (step 1240). The management entity of the management server 101 then delivers the product to the user. Through such a procedure, the VR video providing system 100 can be used to sell products.

[0077] (Variation 3) In addition, the management server 101 can also earn affiliate income by inducing access to the EC site. Specifically, an advertisement displayed on a VR 360-degree video has embedded therein a URL for guiding the EC site and information for identifying the management server 101 (or the administrator) associated with the URL. When the user terminal 102 selects an advertisement via the site access module 315 and connects to an EC server displaying the EC site, this information is notified to the EC server together with the identification information of the management server 101. Upon receiving this notification, the EC server takes steps to pay the management server 101 (or its administrator) a fee for guiding the customer. Furthermore, when a product is purchased on the EC site by the user terminal 102, the EC server takes steps to pay the management server 101 (or its administrator) the price for the product purchased by the guided customer.

[0078] (Variation 4) FIG. 18 is another example of a conceptual diagram 1800 of a VR 360-degree video. As in the first embodiment, this VR 360-degree video can be viewed in all directions with the center of the celestial sphere 1820 as the viewpoint 1810, and the VR 360-degree image projected onto the celestial sphere 1820. Here, VR 360-degree video can be represented by three-dimensional point cloud data that contains basic positional information based on the X, Y, and Z axes, as well as information such as color.

[0079] The cylindrical building 1730 can be considered to exist at a specific location (x4, y4, z5) in the VR space formed in the VR 360-degree video. A specific surface area of ​​the building 1730 is then set as the display area (output area) of the video information 1740, and a three-dimensional object 1850 having the same dimensions, shape, and surface form as this display area is virtually set at a position (x4, y4, z5) in the VR space using a point cloud. The video information 1740 is also expressed by a point cloud. The information output module 314 then displays the video information 1740 in a display area on the surface of the three-dimensional object 1850 based on this point cloud data.

[0080] The point cloud data can be compressed and encoded by a known method in advance. At this time, the point cloud having a three-dimensional structure can be projected onto a bounding box 1840 that surrounds it. The point cloud data may also be divided into patches as needed. Each patch can be treated as a two-dimensional image by placing it in a rectangular texture area defined by a bounding box 1840. When compressing, in addition to the point cloud data, it is also advisable to compress and code the geometry indicating the distance information to the projection plane of the bounding box and the occupancy map indicating where the patch is placed in the rectangular texture area.

[0081] According to such a configuration, the information output module 314 can decode and display, for example, compressed data related to the video information 1740 in real time at a predetermined timing. In addition, since the video information 1740 is expressed as a set of pixels in the VR space by a point cloud, a video can be displayed with high accuracy and quality on a smooth curved surface compared to the case of mapping using a mesh. This makes it possible to output the video information 1740, which has a large amount of data, while suppressing the load.

[0082] (Variation 5) FIG. 21 is an example of a visual image 2100 of VR 360-degree video content. (A) to (C) are parts of the image (e.g., natural view) that is visually recognized when three scenes of VR 360-degree video content are output to an HM display, etc. Examples of images as time passes are shown in the order of (A), (B), and (C). This visual image 2100 includes a cylindrical object 2110 as a target object of the VR 360-degree image, and video information 2120 displayed superimposed on the surface of this cylindrical object 2110. The video information 2120 is an example of insertion information in the present technology.

[0083] The information output module 314 outputs, as a video, video information 2120 in which the present box 2130 is displayed from different angles over time. In other words, the video displayed here is a video shot so as to rotate around the present box 2130. In FIG. 21, the video information 2120 is simplified for convenience, but it can be various information related to the object of the VR 360-degree video or VR 360-degree image, such as an advertising video about a product related to the object existing in the VR space, or an explanatory video of a historical building or the like existing in the VR space.

[0084] The information output module 314 may be configured to play the video information 2120 when the event detection module 316 detects the occurrence of a predetermined event. The predetermined event is not limited to this, but may be, for example, a three-dimensional object existing in the VR space satisfying a predetermined condition. The predetermined condition may be, for example, the appearance of an object existing in the VR space, a predetermined movement of the object, an output time of the VR 360-degree video, etc. This allows the video information 2120 to be effectively output at a timing that is easy for the user to recognize, for example.

[0085] In this modified example, the information output module 314 indicates that it is possible to play yet another video within the VR 360-degree video. It also shows that the information output module 314 can paste another video onto the surface of a three-dimensional object in a VR 360-degree video and play it. The shape of the three-dimensional object and the coordinate information of the surface layer are stored in the VR 360-degree video information 500, and the information such as advertisements to be inserted therein is stored in which three-dimensional position and how it is inserted according to the insertion position information, insertion angle information, and insertion form information. In the example of Figure 5, the insertion position information is described only as x, y, and z coordinate information, but when displaying information on the surface of a three-dimensional object, coordinate information and vector information indicating the three-dimensional structure of the layer on which the display is pasted are also stored.

[0086] In this modification, when an event is detected during playback of a VR 360-degree video, information can be displayed. For example, when the timing stored in the insertion time information of the VR 360-degree video information 500 is reached, a video advertisement can be started to be displayed. In this way, a video can be started to be played at any timing during playback of a VR 360-degree video, and the playback timing of another video can be synchronized in relation to the playback timing of the VR video. For example, the information output module 314 can play a video advertisement for a wedding ring in synchronization with the playback of a scene of a wedding ceremony at a shrine in the VR 360-degree video. The timing for displaying the information (or starting playback of the video) is stored in the insertion time information.

[0087] (Variation 6) FIG. 22 is another example of a visual image 2200 of VR 360-degree video content. (A) to (C) are parts of images (e.g., natural views) that are visually recognized when three scenes of VR 360-degree video content are output to an HM display or the like when the user's viewpoint moves. This visual image 2200 includes a cylindrical object 2210 as the subject of the VR 360-degree image, and video information 2220 displayed superimposed on the surface of this cylindrical object 2210. The video information 2220 is an example of insertion information in the present technology. This VR 360-degree video content shows that the user's viewpoint moves from (A) the front left side of the cylindrical object 2210 to (C) the front side of the cylindrical object 2210 as time progresses.

[0088] The information output module 314 outputs, as a video, video information 2220 in which the dice 2230 is displayed from different angles over time. In this way, the information output module 314 adjusts the output area and direction (posture) of the video information 2220 in response to the movement of the user's line of sight and outputs it. This allows the user to visually recognize the video information 2220 without feeling unnatural, as if it were being played back on the surface of the cylindrical object 2210 in a VR 360-degree video. As a result, it is possible to present predetermined information without impairing the sense of world of the VR 360-degree space.

[0089] Furthermore, using FIG. 22, yet another example of a visual image 2200 of VR 360-degree video content will be described. 22 can also be seen as a diagram illustrating a state in which the user's viewpoint does not move. In this case, the user continues to view the three-dimensional object from the front at the same position. In this case, the information output module 314 displays a moving layer of the video advertisement on the surface of the three-dimensional object 2210 so that the video advertisement moves from the right side to the left side. That is, in this modified example, a video advertisement can be displayed while being moved so as to be superimposed on the surface of a three-dimensional object in a VR 360-degree video.

[0090] (Variation 7) FIG. 23 is another example of a visual image 2300 of a VR 360-degree video. In FIG. 17 of the second embodiment, the information output module 314 displays an icon 1750, which is information indicating that transition to another site is possible, so that the icon can be seen from the beginning so that the user can find it. In contrast, the VR 360-degree video content in Fig. 23 does not display the icon 2350 (an example of predetermined information) so that the user can see it. For example, the icon 2350 is designed as a hidden object. The VR 360-degree video content includes an information display area 2360 on or around the specific object 2330. The information display area 2360 in this example is provided on the surface of the building 1730 so as to surround the video information 2340.

[0091] Information display area 2360 is shown by a dotted line in visual image 2300, but is configured so as not to be visible to the user. Information display area 2360 is configured so that any position of information display area 2360 can be selected by the user operating an input device, for example. Selections in variant example 7 include, but are not limited to, a click operation anywhere in the information display area 2360, a pointing operation, an over-cursor operation (operation of hovering the cursor over the information display area 2360), voice input indicating a selection, etc.

[0092] The input receiving module 313 is capable of receiving a selection operation by the user in the information display area 2360. When the input receiving module 313 receives a selection in the information display area 2360, the information output module 314 outputs an icon 2350 so that the user can visually recognize it. Then, when the user further selects the displayed icon 2350, as in the second embodiment, the input receiving module 313 receives the selection of the icon 2350 (guided link information), and the site access module 315 connects to the internet site to which the user is guided.

[0093] The display condition of the icon 2350, which is a hidden object, is not limited to when the information display area 2360 is selected. For example, the display condition may be that the user performs a predetermined action, such as clapping, shaking a part of the body such as a hand, head, or foot, jumping, etc. Such user movements can be input via a joystick, a smart watch, a mobile device such as a smartphone, or a wearable device.

[0094] In this example, icon 2350 is output on the periphery of object 2330 and on the periphery of video information 2340. However, icon 2350 may be output on the periphery of object 2330, or may be output on top of video information 2340. The visibility of icon 2350 can be improved by outputting icon 2350 on the periphery of object 2330. The user can infer that icon 2350 provides further information about video information 2340 by outputting icon 2350 on top of video information 2340.

[0095] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0096] In the above embodiment, the captured video was converted by VR360-degree video editing software, converted into stereo images or VR360-degree videos, and viewed with a VR head mount. This allows the VR360-degree video to be viewed in a highly immersive environment. However, the VR360-degree video may be configured to be viewed on a smartphone or PC.

[0097] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0098] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. The above-described embodiments disclose at least the configurations described in the claims. [Explanation of symbols]

[0099] 100...VR video providing system, 101...management server, 102...user terminal, 211...management module, 212...execution module, 213...sales module, 213...EC sales module, 311...user management module, 312...VR video output module, 313...input reception module, 314...information output module, 315...site access module

Claims

1. a virtual reality (VR) video output unit that outputs 360-degree VR video; a receiving unit that receives input related to the VR 360-degree video; an information output unit that outputs predetermined information when the receiving unit receives the input; A VR video providing system comprising:

2. The information output unit outputs the predetermined information to a plane or a curved surface in a VR space formed within the VR 360-degree video. The VR video providing system according to claim 1 .

3. The predetermined information is at least one of a description of an object existing in the VR space formed in the VR 360-degree video, a video related to the object, or an advertisement related to the object. The VR video providing system according to claim 1 .

4. The predetermined information includes guide link information that can guide to other Internet sites related to objects existing in the VR space formed in the VR 360-degree video. The VR video providing system according to claim 1 .

5. The predetermined information is an advertisement for a product related to an object existing in the VR space formed within the VR 360-degree video, and includes guide link information that can lead to an e-commerce (EC) site that sells the product. The VR video providing system according to claim 1 .

6. a site access unit that connects to a guided internet site when the reception unit receives the selection of the guide link information; The site access unit displays the internet site to which the user is directed within the window displaying the VR 360-degree video or in a new window different from the window displaying the VR 360-degree video. The VR video providing system according to claim 4.

7. an EC sales unit that displays product information related to an object present in the VR space formed within the VR 360-degree video, accepts selection of the product, and sells the product; The VR video providing system according to claim 1 .

8. The VR 360-degree video has an information display area on or around a specific object, When the reception unit receives a selection in the information display area, the information output unit outputs the predetermined information. The VR video providing system according to claim 1.

9. an event detection unit that detects that a predetermined event has occurred in the VR 360-degree video output by the VR video output unit; When the event detection means detects the occurrence of the predetermined event, the information output unit outputs the predetermined information. The VR video providing system according to claim 1 .

10. The predetermined event is a predetermined action of an object existing in the VR space formed in the VR 360-degree video, or an appearance of the object. The VR video providing system according to claim 9.

11. When the event detection means detects the occurrence of the predetermined event, the information output unit reproduces a video as the predetermined information within the VR 360-degree video. The VR video providing system according to claim 9.

12. When the event detection means detects that a three-dimensional object existing in the VR space formed within the VR 360-degree video satisfies a predetermined condition, the information output unit plays the video on the surface of the three-dimensional object. The VR video providing system according to claim 11.

13. The information output unit plays a video as the predetermined information within the VR 360-degree video based on an output time of the VR 360-degree video. The VR video providing system according to claim 1 .

14. Outputs virtual reality (VR) 360-degree video, Accepting input regarding the VR 360-degree video; outputting predetermined information when the input is received; How to provide VR videos.

15. A program for causing a computer to execute each step of the VR video providing method according to claim 14.

Images