Moving image distribution method, moving image reproduction method, moving image distribution apparatus, and distribution data structure

Abstract

Provided are a moving image distribution method and a moving image distribution apparatus that reduce an increase in server load caused by changes in a user's line of sight. The present invention is characterized by including the following steps: for each of a plurality of viewpoints determined on a celestial sphere with a camera (10) as an observation point, storing an image stream (44) including an image of the celestial sphere; and a distribution step of distributing the image stream (44) to a user terminal (14), wherein, in the distribution step, the image stream (44) of a viewpoint other than the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user terminal (14) is distributed.

Description

Technical Field

[0001] The present invention relates to a moving image distribution method, a moving image reproduction method, a moving image distribution apparatus, and a distribution data structure for distributing moving images. Background Art

[0002] There are known distribution systems for distributing still images and moving images. For example, the distribution system of Patent Document 1 includes a server and a client, and key frame images and differential frame images constituting a moving image to be distributed are stored in a memory of the server. Further, it has the following structure: when the server receives a request from the client, the key frame images and differential frame images stored in the memory are distributed to the client.

[0003] Here, in the panoramic moving image distribution system described in Non-Patent Document 1, there is a server that distributes the entire background as a low-resolution image, and also intercepts and distributes a portion corresponding to the user's line of sight as a high-resolution image. The client that has received the low-resolution image and the high-resolution image can display the portion being viewed by the user with high image quality by synthesizing these images and displaying them on the screen.

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: Japanese Patent No. 6149967 Gazette

[0007] Non-Patent Documents

[0008] Non-Patent Document 1: NTT TechnoCross “Panorama Ultra Player / Panorama Ultra Engine (Panoramic Super Player / Panoramic Super Engine)”, [searched on February 23, 2020], Internet <https: / / www.ntt-tx.co.jp / products / panocho / > Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] However, the above server must perform the process of intercepting high-resolution images in accordance with the constantly changing viewpoints of users. Therefore, when a large number of users access the server, the load on the server will increase accordingly. And if key frame images and differential frame images are generated for the moving images to be transmitted (high image quality moving images) as in Patent Document 1, the load on the server will further increase.

[0011] An object of the present invention is to provide a moving image distribution method, a moving image reproduction method, a moving image distribution apparatus, and a distribution data structure that reduce an increase in server load caused by changes in the user's line of sight.

[0012] Solution to the problem

[0013] To achieve the above object, the moving image distribution method of the present invention includes the following steps: storing an image stream including the celestial sphere for each of a plurality of viewpoints determined on the celestial sphere with a camera as an observation point; and a distribution step of distributing the image stream to a user's terminal, wherein, in the distribution step, the image stream of viewpoints other than the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal is distributed.

[0014] In addition, to achieve the above object, the moving image reproduction method of the present invention includes the following steps: storing an image stream including the celestial sphere for each of a plurality of viewpoints determined on the celestial sphere with a camera as an observation point; and a reproduction step of reproducing the image stream in the user's terminal, wherein, in the reproduction step, the image stream of viewpoints other than the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal is reproduced.

[0015] In addition, to achieve the above object, the moving image distribution apparatus of the present invention includes: a storage unit that stores an image stream including the celestial sphere for each of a plurality of viewpoints determined on the celestial sphere with a camera as an observation point; and a distribution unit that distributes the image stream to a user's terminal, wherein the distribution unit distributes the image stream of viewpoints other than the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal.

[0016] Furthermore, the distribution data structure of the present invention includes an image stream that includes an image on a line of sight projected from a specific observation point in a central portion and an image of the celestial sphere photographed from the observation point outside the central portion, and the image stream includes a first image stream and a second image stream, the first image stream including an image of a viewpoint on a first line of sight projected from the specific observation point in the central portion, and the second image stream including an image of a viewpoint on a second line of sight projected from the observation point in the central portion.

[0017] Effect of the invention

[0018] According to the moving image distribution method, the moving image reproduction method, the moving image distribution apparatus, and the distribution data structure of the present invention, it is possible to reduce an increase in server load caused by changes in the user's line of sight. Description of the drawings

[0019] Figure 1This is a schematic diagram of the moving image distribution system of the present embodiment.

[0020] Figure 2 (a) of is a hardware schematic diagram of the user terminal of the above-mentioned moving image distribution system, (b) is a hardware schematic diagram of the camera of the above-mentioned moving image distribution system, and (c) is a hardware schematic diagram of the server of the above-mentioned moving image distribution system.

[0021] Figure 3 (a) of is a flowchart of the moving image generation and distribution program executed in the above-mentioned server, and (b) is a flowchart of the generation process in the moving image generation program.

[0022] Figure 4 This is a diagram showing the image generated in the generation process.

[0023] Figure 5 This is a diagram showing the position of the viewpoint.

[0024] Figure 6 This is a diagram showing the process of extracting pixels in the video stream generation process.

[0025] Figure 7 This is a diagram showing the correspondence between the viewpoint key frame image and the virtual sphere.

[0026] Figure 8 This is an example diagram showing the function of the viewpoint information.

[0027] Figure 9 This is a diagram showing the correspondence in the case of generating a low image quality part in the viewpoint key frame image.

[0028] Figure 10 This is an example diagram showing a set of video streams by viewpoint.

[0029] Figure 11 This is a flowchart of the user terminal. Detailed implementation mode

[0030] [First Embodiment]

[0031] Hereinafter, a moving image distribution system and a moving image distribution method according to an embodiment of the present invention will be described with reference to the drawings.

[0032] As Figure 1As shown, the moving image distribution system 1 of the first embodiment is a system that distributes moving images (video stream 44) to the user's terminal 14 (hereinafter referred to as the user terminal 14), and includes a camera 10 that generates images, and a server 12 that functions as a distribution device and generates distribution moving images based on the images obtained from the camera 10. These camera 10, server 12, and user terminal 14 are connected to a network represented by an Internet communication line, and the server 12 can communicate with the camera 10 and the user terminal 14.

[0033] The user terminal 14 is, for example, a well-known portable information terminal such as a smart phone or a tablet terminal. As shown in (a) of Figure 2 , it includes: a communication module 16 (communication unit), which is an interface for connecting to an Internet communication line; a liquid crystal display 18 (display unit), which displays the moving images received from the server 12; a touch panel 20 (input unit), which is superimposed on the liquid crystal display 18 and is used to receive inputs from the user; an angular velocity sensor 22 (detection unit), which detects the posture of the terminal; and a CPU 26 (control unit), which controls the liquid crystal display 18, the touch panel 20, and the angular velocity sensor 22 by executing a program stored in the memory 24.

[0034] The camera 10 is a device that generates at least a hemispherical image. As shown in (b) of Figure 2 , it includes: an image sensor 28; a fish-eye lens, which is an optical component that images an image (imaging circle) of a virtual hemispherical surface with an infinite radius centered on the image sensor 28 onto the light-receiving surface of the image sensor 28; a CPU 30, which controls the image sensor 28 and generates a hemispherical image based on the electrical signal output from the image sensor 28; and a communication module 32, which is used to connect to an Internet communication line. The camera 10 generates hemispherical images at a frame rate of 60 fps (frames per second). A plurality of consecutive hemispherical images generated in this way are stored in the memory 34 in the order of the time series in which they are generated. When a plurality of hemispherical images (a set of hemispherical images) generated by shooting within a certain period are accumulated in the memory 34, the camera 10 sends the set of hemispherical images stored in the memory 34 to the server 12 via the Internet communication line.

[0035] The server 12 is a terminal that distributes moving images (video stream 44), which are distribution data generated based on the above set of hemispherical images, to the user terminal 14. As shown in (c) of Figure 2 , it includes a communication module 36 connected to an Internet communication line, a memory 38 storing a moving image generation and distribution program, and a CPU 40 that executes the moving image generation and distribution program.

[0036] like Figure 3 As shown in (a) of FIG. 1 , the motion image generation and distribution program is a program for causing the server 12 to execute the following processes: an acquisition process (s10) of acquiring a set of hemispherical images from the camera 10; a generation process (s20) of generating a video stream 44 for distribution based on the acquired set of hemispherical images; and a distribution process (s30) of distributing the video stream 44 corresponding to the request from the user terminal 14 to the user terminal 14. In this embodiment, the video stream 44 generated in the generation process (s20) is for the video stream 44 as shown in FIG. Figure 5 As shown in FIG. 1 , the image stream 44 is generated for each predetermined viewpoint on a virtual celestial sphere of infinite radius with the image sensor 28 of the camera 10 as the base point. In other words, when the semi-spherical image generated by the camera 10 is mapped on the virtual celestial sphere, the center of the celestial sphere is set to the user's angle, and the viewpoints on the celestial sphere when the celestial sphere is observed from the angle are set to multiple locations, and the image stream 44 is generated for each of the viewpoints in the multiple locations. Then, in the distribution process (s30), the image stream 44 of a viewpoint corresponding to or approximate to the user's line of sight information included in the request from the user terminal 14 is sent to the user terminal 14. The following is a specific explanation.

[0037] The acquisition process (s10) is to acquire a set of hemispherical images 42 ( Figure 1 , Figure 4 ) is processed, and the received set of semi-spherical images 42 are stored in the memory 38 in a time series order. In this way, the server 12 functions as an acquisition unit that acquires a set of semi-spherical images 42 from the camera 10. In addition, the memory 38 of the server 12 functions as a storage unit that stores the set of semi-spherical images 42.

[0038] When the above acquisition process (s10) is executed, the generation process (s20) is executed. Figure 4 As shown, the generation process (s20) is a process for generating an image stream 44 (continuous images continuous in time series) for each predetermined viewpoint based on a set of hemispherical images 42 stored in the memory 38, and includes an intermediate image generation process (s21) and an image stream generation process (s22).

[0039] In the intermediate image generation process (s21), a group of semi-spherical images 42 stored in the memory 38 is extracted, and a group of intermediate images 46 is generated based on the extracted group of semi-spherical images 42 ( Figure 4). The set of intermediate images 46 includes a key frame image 46a and a differential frame image 46b generated by using a known inter-frame prediction. In the present embodiment, the semi-spherical image 42a extracted from the set of semi-spherical images 42 at every predetermined frame (in the present embodiment, every 60 frames) is set as a key frame image. In addition, the differential frame image 46b is generated by finding the difference between the other plurality of semi-spherical images 42b of the frame following the semi-spherical image 42a (key frame image) and the semi-spherical image of the previous frame. The set of intermediate images 46 generated by the intermediate image generation process (s21) is stored in the memory 38 of the server 12. In addition, the semi-spherical image 42 of the last frame in the set of intermediate images 42 is extracted as the key frame image 46a assigned to the last frame in the set of intermediate images 46. In this way, the CPU 40 of the server 12 functions as an intermediate image generation unit that generates a set of intermediate images 46, and the memory 38 of the server 12 functions as a storage unit that stores a set of intermediate images 46. Hereinafter, the key frame image 46a and the difference frame image 46b generated by the intermediate image generation process (s21) are referred to as an intermediate key frame image 46a and an intermediate difference frame image 46b, respectively.

[0040] The image stream generation process (s22) is a process for generating an image stream 44 by viewpoint based on a set of intermediate images 46. The image stream 44 is a continuous image distributed to the user terminal 14, and includes a viewpoint key frame image 44a and a viewpoint difference frame image 44b. As described above, the image stream 44 is generated corresponding to each of the predetermined multiple viewpoints. The predetermined multiple viewpoints are as described above in the image stream including the key frame image 44a and the viewpoint difference frame image 44b. Figure 5 As shown in FIG. 1 , a plurality of points are determined on a virtual celestial sphere of the celestial sphere observed by the image sensor 28 of the camera 10 as an observation point (base point), and each viewpoint is defined by viewpoint information consisting of a roll angle (α), a pitch angle (β), and a yaw angle (γ) with the observation point as the base point. For example, regarding viewpoint a, the viewpoint information is determined as (α a , β a , γ a ), the viewpoint information is stored in the memory 38 in a manner corresponding to the viewpoint identification information assigned to the viewpoint a. In addition, the video stream 44 generated for the viewpoint a is stored in the memory 38 in a manner corresponding to the viewpoint identification information. That is, the viewpoint information of each viewpoint and the video stream 44 generated for each viewpoint are stored in the memory 38 in a manner corresponding to the viewpoint identification information.

[0041] The viewpoint key frame images 44a and the viewpoint difference frame images 44b constituting the video stream 44 are shown in FIG. Figure 6When it is expanded in the user terminal 14 as shown in (a), it is compressed so that the image quality gradually decreases from the center of the image toward the outside. When taking the viewpoint key frame image 44a as an example, as shown in Figure 6 (b), with the center of the viewpoint key frame image 44a as the base point, it is compressed into an image with high image quality inside the inscribed circle that is inscribed in the four sides (edges) of the viewpoint key frame image 44a, and the image quality outside the inscribed circle (the four corners of the image) becomes low. Hereinafter, the generation process of such a viewpoint key frame image 44a will be described taking the viewpoint a as an example.

[0042] As shown in Figure 6 (c) and (d), pixels are extracted from the virtual sphere 56 obtained by virtually mapping the intermediate key frame image 46a to generate the viewpoint key frame image 44a under the viewpoint a (hereinafter referred to as the viewpoint a key frame image 44a). Specifically, as shown in Figure 3 (b), for each pixel that should constitute the viewpoint a key frame image 44a, the corresponding first coordinate on the surface of the virtual sphere 56 is calculated by a correspondence formula (the first calculation process (s221)), and a rotation formula including the viewpoint information of the viewpoint a is applied to this first coordinate to calculate the second coordinate (the second calculation process (s222)), thereby extracting the pixel located at the second coordinate on the surface of the virtual sphere 56. In addition, the coordinates of the viewpoint a key frame image 44a are represented by XY orthogonal coordinates with its center as the origin as shown in Figure 7 (b), and it is set that the horizontal (X coordinate) of the viewpoint a key frame image 44a takes a value of -1 ≤ X ≤ 1, and the vertical (Y coordinate) takes a value of -1 ≤ Y ≤ 1. In addition, the coordinates of the virtual sphere 56 are represented by XYZ orthogonal coordinates with its center as the origin as shown in Figure 7 (a), and the radius r of the virtual sphere 56 is set to 1.

[0043] The above-mentioned first calculation process (s221) includes a spherical coordinate calculation process for obtaining the spherical coordinates (r, θ, φ) in the virtual sphere 56 based on the coordinates of the viewpoint a key frame image 44a and the viewing angle information, and an orthogonal coordinate calculation process for obtaining the orthogonal coordinates (x, y, z) corresponding to the spherical coordinates. In addition, the viewing angle information is information that identifies the range to be displayed on the liquid crystal display 18 of the user terminal 14, and is set to 30° in the present embodiment.

[0044] As shown in Figure 7As shown, taking the pixel P included in the key frame image 44a from the viewpoint a as an example, the spherical coordinate calculation process will be described. The angle θp' with respect to the Z-axis and the angle φp' with respect to the X-axis in the virtual sphere are obtained as follows. In addition, as described above, the radius r of the virtual sphere 56 is 1. The angle θp' is determined based on the distance Pr from the origin to the pixel P in the XY orthogonal coordinates of the key frame image 44a from the viewpoint a and the specified viewing angle information. Regarding the distance Pr, it is determined by the following corresponding formula based on the coordinate values (Px, Py) of the pixel P.

[0045] [Equation 1]

[0046]

[0047] Then, the calculated value of the distance Pr is input into the function f(Pr) determined in advance according to the viewing angle information to obtain the angle θp'. This function is a function that determines the relationship between the distance Pr and the angle θp' as shown in (a) of Figure 8 . For example, when the viewing angle information is set to 30°, the function is determined such that θ is 30° when Pr = 1. Substitute the distance Pr obtained in the above Equation 1 into this function to obtain the angle θ at point P. That is, the function is determined such that the boundary between the high-pixel part and the low-pixel part in the key frame image 44a from the viewpoint a corresponds to the viewing angle information. It can also be as shown in (b) of Figure 8 . When the viewing angle information is 90°, the viewing angle information and the function are determined such that θ is 90° when Pr = 1. In addition, it can be a linear function as shown in (c) of Figure 8 .

[0048] The angle φp' is the same as φp in the XY orthogonal coordinates of the key frame image 44a from the viewpoint a, and this φp is obtained by the following corresponding formula based on the coordinates (Px, Py) of point P.

[0049] [Equation 2]

[0050]

[0051] Here, as shown in Figure 9As shown in (b), when calculating the angle φ for pixels constituting the low image quality portion, such as pixels on the circumference C, in the same manner as the above corresponding formula (Equation 2), the extraction of pixel information with deviation is performed by considering only the pixels corresponding to the arc indicated by the dash-dot line and not considering the pixels on the arc (dashed arc) indicated by the dashed line. Therefore, in the present embodiment, based on the ratio of the dashed arc to the circumference C, the points on the circumference C including the dashed portion are evenly arranged on the dash-dot line, so that the pixel information is extracted in an interlaced manner without deviation, reducing the amount of information in the key frame image 44a (image stream) from the viewpoint a. Therefore, for example, pixel information corresponding to pixel Q' is extracted from pixel Q on the circumference C. The corresponding formula for achieving such uniform arrangement is as follows.

[0052] [Equation 3]

[0053] φ Q′ -45°: Q -45° = 45°: 45° - φ i

[0054]

[0055] Here, φ i is the angle used to calculate the ratio (proportion) of the dashed arc to the circumference C.

[0056] When calculating the spherical coordinates (1, θ, φ) for each pixel in the key frame image 44a from the viewpoint a as described above, in the orthogonal coordinate calculation process, the first coordinates (x1, y1, z1) for each pixel are calculated using the following transformation formula.

[0057] [Equation 4]

[0058] x1 = r × sinθ × cosφ

[0059] [Equation 5]

[0060] y1 = r × sinφ × cosθ

[0061] [Equation 6]

[0062] z1 = r × cosθ

[0063] When the orthogonal coordinate calculation process is executed, the second calculation process is then executed. In the second calculation process, a rotation formula including viewpoint information (α a , β a , γ a ) is applied to each first coordinate to calculate the second coordinates (x2, y2, z2).

[0064] [Equation 7]

[0065]

[0066] The pixels to be extracted in the virtual sphere are determined by the above second calculation process. Then, the information of the determined pixels is extracted, and the information of the extracted pixels is assigned to the corresponding pixels in the viewpoint a key frame image 44a. In this way, the viewpoint a key frame image 44a obtained by the following method is generated: within the inscribed circle formed as the high image quality part, pixels on the virtual sphere are extracted in a fisheye image shape corresponding to the viewing angle, and outside the inscribed circle formed as the low image quality part, pixels on the virtual sphere outside the viewing angle are extracted in an intermittent manner.

[0067] As described above, the generation process of the per-viewpoint key frame image 44a under viewpoint a has been described. However, the per-viewpoint differential frame image 44b under viewpoint a is also generated by the same process. In this way, the video stream 44 of viewpoint a is generated. For other viewpoints, the video stream 44 (the per-viewpoint key frame image 44a and the per-viewpoint differential frame image 44b) is also generated by the same process as that of viewpoint a, and the generated video stream 44 is stored in the memory 38 of the server 12 in a manner associated with the viewpoint information (associated with the viewpoint information by corresponding to the viewpoint identification information). In this way, the memory 38 of the server 12 functions as a storage unit that stores the video streams 44 of each viewpoint in a manner corresponding to the viewpoint information.

[0068] The video streams 44 of each viewpoint are generated as described above. However, in the present embodiment, the per-viewpoint key frame images 44a constituting the video stream 44 are not synchronized between viewpoints. The per-viewpoint key frame image 44a of one viewpoint and the per-viewpoint key frame images 44a of other viewpoints are arranged at different times in a time series and stored in the memory 38. That is, each video stream 44 is arranged in such a way that the per-viewpoint key frame image 44a and the per-viewpoint differential frame image 44b are not synchronized with each other in a time series. For example, as Figure 10 shown, in the video streams 44 of viewpoints a to d, the per-viewpoint key frame images KF002a, KF002b, KF002c, and KF002d of each of the viewpoints a to d are images generated based on the intermediate key frame image KF002. However, the viewpoint b key frame image KF002b is arranged to be 4 frames delayed relative to the viewpoint a key frame image KF002a, the viewpoint c key frame image KF002c is arranged to be 9 frames delayed relative to the viewpoint a key frame image KF002a, and the viewpoint d key frame image KF002d is arranged to be 14 frames delayed relative to the viewpoint a key frame image KF002a. For example, in the video stream 44 of viewpoint b, the viewpoint b key frame image KF001b (the first per-viewpoint key frame image 44a) is continuously arranged from frame 1 to frame 4, so that the video streams 44 of each viewpoint are not synchronized with each other as described above.

[0069] Next, the distribution process (S30) for the user terminal 14 will be described.

[0070] Before the distribution process (S30), the server 12 establishes a peer-to-peer connection with the user terminal 14 through a signaling server 12 (not shown) so that they can communicate with each other and receive a request from the user terminal 14 (S40)( Figure 11 ). This request is a request for the server 12 to distribute information on a moving image, and includes the line-of-sight information of the user terminal 14. The line-of-sight information is information that identifies the line of sight of the user (the center of the image to be displayed on the user terminal 14), and includes the roll angle (α), pitch angle (β), and yaw angle (γ) determined by the CPU 26 of the user terminal 14 based on the output signal of the angular velocity sensor 22.

[0071] When the server 12 receives a request from the user terminal 14, it compares the line-of-sight information included in the request with a plurality of viewpoint information stored in the memory, and distributes the video stream 44 corresponding to the viewpoint information that is the same as or approximate to the line-of-sight information to the user terminal 14.

[0072] As Figure 11 shown, when the user terminal 14 receives the video stream 44, it performs an unfolding process (S60). In the unfolding process (S60), first, key frame images and differential frame images for unfolding are generated based on the received video stream 44. The pixels in the high image quality portion of the per-viewpoint key frame image 44a are arranged as they are in the central portion of the key frame image for unfolding. Around this high image quality portion, the images of the low image quality portion of the per-viewpoint key frame image 44a are arranged. Here, regarding the corner pixels of this low image quality portion, instead of arranging them as they are, the position of φ Q ’ is determined using the above-mentioned formula 4, and pixels are arranged at the determined positions. At this time, since pixels are not continuously arranged on the circumference C including φ Q ’, an interpolation process for interpolating between each pixel is performed. This interpolation process is not particularly limited. For example, pixels approximate to each pixel are arranged between the pixels existing on the same circumference. Through the same process as this key frame image for unfolding, a differential frame image for unfolding is generated.

[0073] When the key frame image for unfolding and the differential frame image for unfolding are generated through this interpolation process, a key frame image for display and a differential frame image for display are generated using a known panoramic unfolding process, and a moving image is generated based on these images and displayed on the user terminal 14.

[0074] While the moving image is being displayed (played back) on the user terminal 14, the user terminal 14 also monitors the user's line of sight by confirming the output of the angular velocity sensor 22 through its CPU 26, and the display coordinates of the moving image are displaced according to the change amount of the line of sight. In addition, the user terminal 14 updates the line-of-sight information and sends this line-of-sight information to the server 12.

[0075] Whenever the server 12 receives the line-of-sight information, it extracts the viewpoint information of the video stream 44 in which the key frames are configured to be close in time series, compares the received line-of-sight information with the extracted viewpoint information to retrieve the most similar viewpoint, and sends the video stream 44 corresponding to the similar viewpoint to the user terminal 14.

[0076] Here, during the period when the user's line of sight changes from line of sight a to line of sight f, specifically, when the posture of the user terminal 14 changes due to the terminal operation performed by the user, and during the period when the user's line of sight detected based on this posture change changes from viewpoint a to viewpoint f, the video stream 44 is distributed as follows.

[0077] When the server 12 receives the line-of-sight information from the user terminal 14, it uses the relationship with the received time point to retrieve the video stream 44 at the moment when the viewpoint key frame image 44a is configured to be close in time series. Specifically, since each video stream 44 stored in the memory 38 is generated in a manner that is out of sync with each other in time series as described above, the configuration position (configuration moment) of the viewpoint key frame image 44a in the video stream 44 is different in multiple video streams 44. The CPU 40 of the server 12 calculates the configuration position (configuration moment) of the key frame image based on the key frame period (60 frames) in each video stream 44 and the delay amount set for each video stream 44, and retrieves the video stream 44 having the viewpoint key frame image 44a at the moment closest to the frame image being distributed at the time point when the line-of-sight information is received (the frame image being played back on the user terminal 14).

[0078] Then, it is determined whether the viewpoint information corresponding to the retrieved video stream 44 is closer in position to the changed viewpoint (viewpoint f) than the viewpoint before the change (viewpoint a).

[0079] For example, in the case where the result of retrieving the viewpoint key frame image 44a determines that the viewpoint key frame image 44a of viewpoint c is close in time series, viewpoint c is closer in position to viewpoint f than viewpoint a, so the video stream 44 of viewpoint c is distributed to the user terminal 14.

[0080] On the other hand, even when the result of retrieving the viewpoint key frame image 44a by viewpoint determines that the viewpoint key frame image 44a of viewpoint g is arranged at a time close in time series, since viewpoint g is farther from viewpoint f in position than viewpoint a, the video stream 44 of viewpoint a is distributed to the user terminal 14.

[0081] In the moving image distribution system 1 of the present embodiment, video streams 44 corresponding to a plurality of viewpoints are generated in advance. Therefore, even when the user's line of sight is generated, it is only necessary to distribute the video stream 44 corresponding to the viewpoint obtained based on the line of sight. Therefore, even if there are requests from a large number of user terminals 14, for example, an increase in the load on the server 12 can be reduced.

[0082] In addition, even if the display coordinates of the moving image being displayed are displaced as a result of a change in the posture of the user terminal 14, causing the image quality to gradually deteriorate, since the video stream 44 having viewpoint key frame images 44a close in time series and position is distributed, a significant decrease in the image quality of the image being displayed can be prevented.

[0083] [Second Embodiment]

[0084] In the above first embodiment, it is assumed that the server 12 selects the video stream 44 to be distributed based on the line-of-sight information received from the user terminal 14. However, in the second embodiment, it is assumed that the user terminal 14 selects the video stream 44 to be received based on the line-of-sight information and requests the server 12 to distribute the selected video stream 44. Hereinafter, the description will focus on the structure and process different from the first embodiment, and the same structure and method as the first embodiment will be appropriately omitted.

[0085] In the present embodiment, similar to the first embodiment, the video stream 44 generated by viewpoint is stored in the memory of the server 12 in a manner corresponding to the viewpoint identification information. However, the difference is that the viewpoint information is not stored in the memory 38 of the server 12. In the present embodiment, each piece of viewpoint information is stored in the memory 24 of the user terminal 14 in a manner corresponding to the viewpoint identification information.

[0086] In addition, in the present embodiment, similar to the first embodiment, the plurality of image streams 44 generated for each viewpoint include the key-frame images 44a for each viewpoint. Moreover, among these plurality of image streams 44, the key-frame images 44a for each viewpoint are arranged out of synchronization with each other in time series by shifting the first key-frame image 44a for each viewpoint. In the present embodiment, the arrangement time of the key-frame images 44a for each viewpoint in the image stream 44 for each viewpoint is stored in the memory 24 of the user terminal 14. This arrangement time indicates at which time (frame) the key-frame image 44a for each viewpoint is arranged in the image stream 44 for each viewpoint. Typically, it represents the interval (arrangement period) of the key-frame images 44a for each viewpoint in each image stream 44, and the number of shifts (number of delayed frames) of the first key-frame image 44a for each viewpoint in the image stream 44 for each viewpoint. In the present embodiment, as Figure 4 shown, the key-frame images for each viewpoint are arranged every 60 frames, so the interval is "60". Additionally, as Figure 10 shown, there is no shift in the key-frame image 44a for viewpoint a, so the number of shifts for viewpoint a is "0". Additionally, in the image stream 44 for viewpoint b, a shift of 4 frames is set for the first key-frame image 44a, so the number of shifts for viewpoint b is "4". Similarly, the number of shifts for viewpoint c is "9", and the number of shifts for viewpoint d is "14". By storing the defined arrangement times for each viewpoint in a manner corresponding to the viewpoint identification information in this way, the viewpoint identification information is stored respectively in a manner corresponding to the viewpoint information.

[0087] The user terminal 14 of the present embodiment determines the image stream 44 for the viewpoint to be received based on the line-of-sight information as described above, and requests the server 12 to distribute the determined image stream 44 for the viewpoint. Specifically, the user terminal 14 executes the line-of-sight information acquisition process, the request process, and the display process in this order.

[0088] (1) The line-of-sight information acquisition process is a process in which the CPU 26 of the user terminal 14 acquires the line-of-sight information based on the output from the angular velocity sensor 22, and acquires the roll angle (α), the pitch angle (β), and the yaw angle (γ) in the same way as in the first embodiment.

[0089] (2) The request process is to extract the viewpoint information approximate to the line-of-sight information acquired in the above line-of-sight information acquisition process, and send the viewpoint identification information corresponding to the extracted viewpoint information to the server 12. When the server 12 receives the viewpoint identification information from the user terminal 14, it distributes the image stream 44 corresponding to the viewpoint identification information to the user terminal 14.

[0090] (3) The display process is a process of displaying the image stream 44 on the liquid crystal display 18 while receiving the image stream 44 from the server 12.

[0091] The distribution of the video stream 44 and the display of the video stream 44 in the initial stage are performed through the above process.

[0092] As described above, while displaying the video stream 44, the CPU 26 of the user terminal 14 performs a line-of-sight information acquisition process, a determination process, a request process, and a display process in synchronization with the frame rate of the video stream 44 to display a video stream 44 corresponding to a line-of-sight change caused by a user operating the user terminal 14.

[0093] (4) The line-of-sight information acquisition process is the same as the process in (1) above, and is a process of acquiring line-of-sight information (roll angle (α), pitch angle (β), and yaw angle (γ)) based on the output of the angular velocity sensor 22.

[0094] (5) The determination process is a process of determining the video stream to be requested from the server 12. The CPU 26 of the user terminal 14 selects viewpoint identification information configured in the viewpoint key frame images 44a that are close in time series.

[0095] (5-1) Specifically, the frame number in the video stream 44 being reproduced (hereinafter referred to as the reproduction frame number) is determined. For example, when the 100th frame image is being displayed while reproducing the video stream 44 of viewpoint a, the frame number is determined as "100".

[0096] (5-2) Next, based on the determined interval and offset of the configuration time, the configuration positions of the viewpoint key frame images 44a are calculated for each viewpoint, and the numbers of the key frames configured after the determined frame number and close to the determined frame number in time series are extracted.

[0097] For example, regarding the configuration time of viewpoint b, the interval is "60" and the offset is defined as "4", so the position of the first viewpoint key frame image 44a of viewpoint b is calculated as "5", the position of the second viewpoint key frame image 44a is calculated as "65", the position of the third viewpoint key frame image 44a is calculated as "125", and the position of the fourth viewpoint key frame image 44a is calculated as "185". Moreover, whenever the positions of these viewpoint key frame images 44a are calculated, the difference from the determined frame number "100" is calculated, and the position of the viewpoint key frame image 44a with the least difference, specifically the position "124" of the third viewpoint key frame image, is approximated to the determined frame number "100".

[0098] Calculations are similarly performed for viewpoints c, d, etc. For viewpoint c, the position "129" of the third viewpoint key frame image is set to be approximate to the determined frame number "100". Additionally, for viewpoint d, the position "74" of the second viewpoint key frame image is the closest, but since it is before the determined frame number "100", the position "134" of the next approximate third viewpoint key frame image is set to be approximate to the determined frame number "100".

[0099] In this way, when the positions of the viewpoint key frame images 44a approximate to each viewpoint are calculated, the viewpoint that is the closest to the determined frame number is selected. In the above example, viewpoint b that is approximate to the determined frame number "100" is selected.

[0100] (5-3) When the viewpoint closest to the viewpoint key frame image 44a is selected (viewpoint b in the above example), the distance between this viewpoint (viewpoint b) and the line-of-sight information is calculated. Additionally, the distance between the viewpoint being reproduced (viewpoint a) and the line-of-sight information is calculated. Then, the viewpoint with the shorter of the two distances is determined as the viewpoint to be reproduced, and the viewpoint identification information corresponding to this viewpoint is extracted. That is, when the currently reproduced viewpoint (viewpoint a) is approximate to the line-of-sight information, the currently reproduced viewpoint (viewpoint a) is continuously requested. On the other hand, when the line-of-sight information is closer in coordinates to the viewpoint (viewpoint b) where the viewpoint key frame is configured at a time close to the currently reproduced frame than the currently reproduced viewpoint (viewpoint a), the video stream 44 of this viewpoint (viewpoint b) is newly requested.

[0101] (6) In the request processing, this viewpoint identification information and the determined frame number are sent to the server 12. When the server 12 receives the viewpoint identification information and the frame number, it sends the video stream 44 corresponding to this viewpoint identification information and starting from the frame image corresponding to the determined frame number to the user terminal 14.

[0102] (7) The user terminal 14 displays the position corresponding to the line-of-sight information of the received video stream 44 at the center of the liquid crystal display 18.

[0103] [Third Embodiment]

[0104] The user terminal 14 in the above-described first and second embodiments is a reproduction mode that receives and reproduces the video stream 44 distributed from the server 12, but is not limited to this reproduction mode. The third embodiment does not include the server 12, and the video stream 44 generated according to the viewpoints is stored in the memory 24 of the user terminal 14 in a manner corresponding to the viewpoint information. In addition, the viewpoint key frame images 44a at the beginnings of these multiple video streams 44 are offset in the same manner as in the first and second embodiments, so that the viewpoint key frame images 44a are configured to be out of sync with each other in time series. Moreover, the configuration times of the viewpoint key frame images 44a in the video streams 44 of each viewpoint are stored in the memory 24 of the user terminal 14.

[0105] In such an embodiment of the present invention, in the user terminal 14, the line-of-sight information acquisition process and the reproduction process are executed in this order.

[0106] (1) The line-of-sight information acquisition process is a process in which the CPU 26 of the user terminal 14 acquires line-of-sight information based on the output from the angular velocity sensor 22, and acquires the roll angle (α), pitch angle (β), and yaw angle (γ) in the same manner as in the first and second embodiments.

[0107] (2) In the reproduction process, viewpoint information whose value is approximate to the line-of-sight information acquired in the above line-of-sight information acquisition process is extracted, and the video stream 44 corresponding to the extracted viewpoint information is reproduced.

[0108] As described above, the CPU 26 of the user terminal 14 reproduces the video stream 44 while executing the line-of-sight information acquisition process and the reproduction process in synchronization with the frame rate of the video stream 44 to reproduce the video stream 44 corresponding to the change in the line of sight caused by the user's operation on the user terminal 14.

[0109] (4) The line-of-sight information acquisition process is the same as the process in (1) above, and is a process of acquiring line-of-sight information (roll angle (α), pitch angle (β), and yaw angle (γ)) based on the output from the angular velocity sensor 22.

[0110] (5) The reproduction process is a process of determining the video stream 44 to be reproduced and reproducing it. It is assumed that the CPU 26 of the user terminal 14 selects the viewpoint information in which the viewpoint key frame image 44a is arranged near the reproduction time point in time series, and selects the video stream 44 to be reproduced based on the selected viewpoint information and the line-of-sight information acquired in (4) above.

[0111] (5-1) Specifically, the frame number in the video stream 44 being reproduced (hereinafter referred to as the reproduction frame number) is determined.

[0112] (5-2) Next, for each video stream 44 of the video streams 44 by viewpoint stored in the memory 24, the configuration position of the key frame image 44a by viewpoint is calculated based on the configuration time. Then, for each video stream 44 of the video streams 44 by viewpoint, the number of frames up to the calculated configuration position of the key frame image is calculated. That is, the number of frames from the reproduction frame number to the configuration position of the key frame image is counted, the video stream 44 of the viewpoint where the key frame image with the smallest count value is located is determined, and the viewpoint information corresponding to the video stream 44 of this viewpoint is extracted.

[0113] (5-3) Next, the distance between the extracted viewpoint information and the line-of-sight information obtained in (4) above is calculated. In addition, the distance between the viewpoint information of the video stream 44 being reproduced and the line-of-sight information is calculated. Then, the viewpoint with the shorter of the two distances is determined as the viewpoint to be reproduced, and the video stream 44 corresponding to this viewpoint is reproduced.

[0114] That is, when the viewpoint information of the video stream 44 currently being reproduced is approximate to the user's line-of-sight information, the video stream 44 currently being reproduced is continued to be reproduced. On the other hand, when the viewpoint information at the time when the key frame image 44a by viewpoint is configured close to the currently reproduced frame is closer to the user's line-of-sight information in terms of coordinates than the currently reproduced viewpoint information, the video stream 44 of this viewpoint information is newly reproduced.

[0115] The present invention is not limited to the above-described embodiments, and may also be in the following manner.

[0116] <Modification Example 1>

[0117] In the above-described embodiment, it is assumed that the video stream 44 by viewpoint is generated based on a set of hemispherical images captured by the camera 10, but the video stream by viewpoint may also be generated based on a set of full-spherical images captured by the camera 10. In addition, not limited to the hemispherical and full-spherical, it may also be a virtual spherical surface with an infinite radius observed with a camera 10 having a viewing angle of 45 degrees as the observation point. In this way, the present invention only needs to generate the video stream by viewpoint based on a set of celestial sphere images captured by the camera 10.

[0118] <Modification Example 2>

[0119] In the above-described embodiment, the camera 10 that captures the real world is used, but it may also be a camera that captures the virtual world.

[0120] <Modification Example 3>

[0121] In the above-described embodiment, the acquisition process and the generation process of the server 12 are not essential processes. Before distributing to the user terminal 14, it is also possible to prepare video streams 44 in advance corresponding to multiple viewpoints and store them in the memory 38.

[0122] <Modification Example 4>

[0123] In the above-described embodiment, the video streams 44 are arranged such that the viewpoint key frame images 44a are out of sync in time series between the viewpoints. However, for example, it is also possible to make them out of sync by continuously arranging a plurality of blank images or the like that are not related to the viewpoint key frame images and the viewpoint differential frames at the beginning of the video stream. In addition, it can also be configured such that the arrangement intervals of the viewpoint key frame images in the video streams between the viewpoints are different. For example, the viewpoint key frame images of viewpoint a are arranged every 60 frames, the viewpoint key frame images of viewpoint b are arranged every 55 frames, the viewpoint key frame images of viewpoint c are arranged every 50 frames, and the viewpoint key frame images of viewpoint d are arranged every 45 frames, thereby making them out of sync in time series.

[0124] <Modification Example 5>

[0125] In the above-described second embodiment, since the interval of the key frames is fixed, the arrangement list is defined according to the interval value and the offset value. However, it is not limited to such a method. For example, when the interval of the key frames is random, it is also possible to store the position (number) list of the key frames in each video stream corresponding to the viewpoint identification information.

[0126] <Modification Example 6>

[0127] In the above-described embodiment, it is configured such that when the user's line of sight changes due to an operation on the user terminal 14 by the user, the video stream 44 including the viewpoint key frame image 44a at the moment closest in time series is selected as the distribution object or the reproduction object. However, it is not limited to the closest moment, and it is also possible to select the video stream 44 including the viewpoint key frame image 44a at a close moment.

[0128] Here, the "close moment" means: with respect to the video stream 44 of the viewpoint that is closest in position to the viewpoint of the video stream 44 being distributed or reproduced (the video stream 44 of the closest viewpoint), based on the frame count from the distribution moment (reproduction moment) to the key frame image of the video stream 44 of this closest viewpoint, it is less than the frame count serving as this benchmark. That is, in each of the video streams 44 of the video streams 44 by viewpoint, the arrangement position (arrangement moment) of the key frame image 44a by viewpoint is calculated. When the arrangement position (arrangement moment) of the key frame image 44a by viewpoint before and after the distribution moment (reproduction moment) is less than the benchmark, the video stream 44 including the key frame image 44a by viewpoint is selected as the moment close in time series, and the viewpoint information of the selected video stream 44 is compared with the changed line-of-sight information.

[0129] In addition, the above frame count serving as the benchmark can also be the frame count obtained by subtracting the number of frames considering the distribution delay caused by the network environment.

[0130] Explanation of reference numerals

[0131] 1: Moving image distribution system; 10: Camera; 12: Server; 14: User terminal; 28: Image sensor (imaging unit); 30: CPU; 34: Memory; 38: Memory; 40: CPU.

Claims

1. A method for distributing moving images, characterized in that, It includes the following steps: A storage step of storing a plurality of video streams generated for each of a plurality of viewpoints determined on the celestial sphere with a camera as an observation point; And A distribution step of distributing the video streams to the user's terminal, wherein each of the video streams generated for each of the viewpoints includes an image of the entire celestial sphere in such a manner that an image of the viewpoint is included in the central part, in the distribution step, when receiving the line-of-sight information determined by the user's terminal, distributing the video stream of the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal; in the distribution step, when the line of sight changes, distributing the video stream of a viewpoint that is closer in position to the viewpoint corresponding to the line of sight after the change than to the viewpoint corresponding to the line of sight before the change; the video stream includes key frame images and differential frame images, the key frame images of one viewpoint are not synchronized in time series with the key frame images of other viewpoints, in the distribution step, when the line of sight changes, if the viewpoints where the key frame images are arranged closely in time series are closer in position to the viewpoint corresponding to the line of sight after the change than to the viewpoint corresponding to the line of sight before the change, distributing the video stream of the viewpoints where the key frame images are arranged closely in time series.

2. The moving image distribution method according to claim 1, characterized in that in the video streams of the other viewpoints, a plurality of consecutive key frame images are arranged.

3. The moving image distribution method according to claim 1, characterized in that in the video streams of the other viewpoints, the first key frame image is arranged in a manner that is delayed compared to the first key frame image in the video stream of the one viewpoint.

4. The moving image distribution method according to claim 1, characterized in that the arrangement interval of the key frame images in the video streams of the other viewpoints is different from the arrangement interval of the key frame images in the video stream of the one viewpoint.

5. A moving image distribution device, characterized in that, It comprises: a storage unit that stores a plurality of video streams generated for each of a plurality of viewpoints determined on the celestial sphere with a camera as an observation point; and a distribution unit that distributes the video streams to the user's terminal, wherein each of the video streams generated for each of the viewpoints includes an image of the entire celestial sphere in such a manner that an image of the viewpoint is included in the central part, the distribution unit distributes the video stream of the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal when receiving the line-of-sight information determined by the user's terminal, the distribution unit distributes the video stream of a viewpoint that is closer in position to the viewpoint corresponding to the line of sight after the change than to the viewpoint corresponding to the line of sight before the change when the line of sight changes, the video stream includes key frame images and differential frame images, the key frame images of one viewpoint are not synchronized in time series with the key frame images of other viewpoints, When the line of sight changes, if the viewpoints at which the key frame images are arranged closely in the time series are closer in position to the viewpoint corresponding to the line of sight after the line of sight change than to the viewpoint corresponding to the line of sight before the line of sight change, the video stream of the viewpoints at which the key frame images are arranged closely in the time series is distributed.

6. A method for reproducing a moving image, characterized in that, including the following steps: storing a plurality of video streams generated for each of a plurality of viewpoints determined on the celestial sphere with the camera as the observation point; and a reproduction step of reproducing the video stream in the user's terminal, wherein each of the video streams generated for each of the viewpoints includes an image of the entire celestial sphere in such a manner that the image of the viewpoint is included in the central part, in the reproduction step, when the line of sight information determined by the user's terminal is received, the video stream of the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal is reproduced, in the reproduction step, when the line of sight changes, the video stream of the viewpoint that is closer in position to the viewpoint corresponding to the line of sight after the line of sight change than to the viewpoint corresponding to the line of sight before the line of sight change is reproduced, the video stream includes key frame images and differential frame images, the key frame images of one viewpoint are out of sync in the time series with the key frame images of other viewpoints, in the reproduction step, when the line of sight changes, if the viewpoints at which the key frame images are arranged closely in the time series are closer in position to the viewpoint corresponding to the line of sight after the line of sight change than to the viewpoint corresponding to the line of sight before the line of sight change, the video stream of the viewpoints at which the key frame images are arranged closely in the time series is reproduced.

7. A distribution data structure having a plurality of video streams, the plurality of video streams including an image on the line of sight projected from a specific observation point in the central part and an image of the entire celestial sphere photographed from the observation point outside the central part, the plurality of video streams include a first video stream and a second video stream, the first video stream includes an image of a viewpoint on a first line of sight projected from a specific observation point in the central part, the second video stream includes an image of a viewpoint on a second line of sight projected from the observation point in the central part, the first video stream and the second video stream each include key frames, the key frames of the first video stream are out of sync in the time series with the key frames of the second video stream, the distribution data structure includes the time information of the key frames in the first video stream and the time information of the key frames in the second video stream, when distributing the distribution data structure, when the line of sight information determined by the user's terminal is received, the video stream of the nearest viewpoint on the celestial sphere corresponding to the line of sight determined by the user's terminal is distributed, when the line of sight changes, the video stream of the viewpoint that is closer in position to the viewpoint corresponding to the line of sight after the line of sight change than to the viewpoint corresponding to the line of sight before the line of sight change is distributed. When the line of sight changes, in a case where viewpoints that are arranged closely in time series of key frames are closer in position to a viewpoint corresponding to the line of sight after the line of sight change than to a viewpoint corresponding to the line of sight before the line of sight change, a video stream of the viewpoints that are arranged closely in time series of the key frames is distributed.

8. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 4 and 6.

Images