Image region generation system and program, and image region display device

Abstract

To enable a moving image imaged by an omnidirectional imaging device to be transmitted into a virtual space at high speed and displayed with a sense of presence.SOLUTION: Each still image that makes up an acquired moving image is cut into a plurality of image regions in accordance with positional relationships of each surface. The cut image regions are assigned to the respective surfaces. Data including the assigned image regions are transmitted on different channels than one another to each display device that is for displaying an image region on each surface. Here, an adjustment is performed for achieving chronological synchronization among the image regions of the transmitted data.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image area generating system, a program, and an image area display device that generate image areas to be displayed on each of rectangular faces surrounding a space. [Background technology]

[0002] In recent years, services that allow users to freely move around in a three-dimensional virtual space constructed online, where the characters (avatars) they control themselves, are becoming more and more popular. These services allow users to enjoy entertainment such as games and sightseeing, as well as economic activities such as buying and selling goods and services, and allow users to engage in a variety of activities in the virtual space as a living space. In particular, with the advancement of VR (Virtual Reality) and AR (Augmented Reality) technologies, users are now able to experience virtual spaces that are closer to reality, and it is expected that the demand for these services will increase rapidly in the future.

[0003] To use a service that utilizes a virtual space, a user wears a glasses- or goggle-type head-mounted image display device on his / her head. Such a head-mounted image display device has a built-in motion sensor, microphone, etc., and can freely switch the image displayed according to the user's motion and voice. This allows the user to freely move an avatar in the virtual space and move their line of sight via the head-mounted image display device, and enjoy a variety of services.

[0004] However, the above-mentioned conventional services using virtual space require users to wear a head-mounted image display device every time, and there was a need to respond to requests to reduce the sense of pressure and inconvenience associated with wearing the device, as well as the effort required for wearing the device. There was also a problem with the effects on the body, such as so-called VR sickness, caused by a discrepancy between the information obtained visually through the head-mounted image display device and the information received by the real body. In addition to this, there was a growing demand for multiple people to simultaneously share and watch omnidirectional images in one virtual space, rather than having each user wear a head-mounted image display device and watch images independently.

[0005] Conventionally, a method for displaying omnidirectional images in a virtual space without wearing a head-mounted image display device has been proposed, for example, as shown in Patent Document 1. However, there is no mention of a specific method for transmitting various moving image contents and moving images captured by an omnidirectional imaging device into a virtual space at high speed and displaying them with a sense of realism. In addition, the surfaces that constitute the available virtual space are configured with various aspect ratios, but the current situation is that no technology has yet been proposed for flexibly creating image areas with a sense of realism according to the shape of the virtual space configured with surfaces from such various size ratios. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-177587 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, the present invention has been devised in consideration of the above-mentioned problems, and its object is to provide an image area generating system, a program, and an image area display device that can generate an image area to be displayed on each rectangular surface surrounding a space, without requiring the user to wear a head-mounted image display device each time, and that can allow a plurality of people to simultaneously share and view an omnidirectional image in one virtual space, and that can transmit a moving image captured by an omnidirectional imaging device at high speed into the virtual space and display it with a sense of presence. Also, the present invention has an object to provide an image area generating system, a program, and an image area display device that can flexibly generate an image area with a sense of presence according to the shape of a virtual space composed of surfaces with various size ratios. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the inventors have invented an image area generation system and program, and an image area display device, which cut out each still image that constitutes an acquired moving image into a plurality of image areas in accordance with the positional relationship of each of the above-mentioned surfaces, assign each of the cut-out image areas to each of the above-mentioned surfaces, and transmit data containing each of the assigned image areas to each display device for displaying the image areas on each surface over different channels, and make adjustments to achieve chronological synchronization between each of the image areas of the data to be transmitted.

[0009] The image area generation system of the first invention is an image area generation system which generates an image area to be displayed on each of rectangular surfaces surrounding a space, and is characterized in comprising: a moving image acquisition means for acquiring a moving image and distribution destination information to which the moving image is to be distributed; a discrimination means for determining the aspect ratio between the surfaces based on an image of each surface captured by an imaging device installed in the space; an image area cut-out means for cut-out a still image to be displayed into a plurality of image areas to be allocated to each surface based on the aspect ratio between the surfaces determined by the discrimination means and the distribution destination information acquired by the moving image acquisition means; and an allocation means for allocating each image area cut out by the image area cut-out means to each surface.

[0010] The image area generation system of the second invention is characterized in that, in the first invention, the moving image acquisition means acquires the distribution destination information including one or more of distributable contract information, distributable facility information, and customer information regarding distributable spectators.

[0011] The image area generation system of the third invention is characterized in that, in the first invention, the video image acquisition means acquires the distribution destination information including one or more of management information for each video or audio information, distribution request information, ranking information, and past playback history information.

[0012] An image area generation system according to a fourth aspect of the present invention is an image area generation system that generates an image area to be played on each of rectangular faces surrounding a space, the image area generation system including: a video acquisition means that acquires at least one of a live video and an archive video, audio information corresponding to the video, and distribution destination information for distributing the video and the audio information; a determination means that determines an aspect ratio between the faces based on an image of each face captured by an imaging device installed in the space; and a determination means that determines, for each still image constituting the video acquired by the video acquisition means, an area to be assigned to each face based on the positional relationship between the faces, the distribution destination information, and the aspect ratio between the faces determined by the determination means. The image processing system is characterized in that it comprises an image area cutting means for cutting out a plurality of image areas, an allocation means for determining characteristics of each image area cut out by the image area cutting means and characteristics of the space, and for allocating each of the image areas to each of the surfaces based on the determined characteristics of each image area and characteristics of the space, and for allocating the acoustic information based on each of the allocated image areas, and a data transmission means for transmitting data including at least either of the image areas allocated by the allocation means or the acoustic information to each reproduction device including at least either of the display devices for reproducing the image areas on each of the surfaces, or the acoustic devices for reproducing the acoustic information, via mutually different channels.

[0013] The image area generation system of the fifth invention is characterized in that, in the fourth invention, the image area cut-out means cuts out each still image constituting the moving image into a plurality of image areas in accordance with the positional relationship of each of the faces based on the acoustic information.

[0014] The image area generating system of the sixth invention is characterized in that, in the fourth invention, the data transmitting means performs adjustments to achieve chronological synchronization between the image areas and acoustic information of the data to be transmitted.

[0015] An image area generation system according to a seventh aspect of the present invention is an image area generation system which generates an image area to be played on each of rectangular surfaces surrounding a space, the image area generation system comprising: video acquisition means for acquiring at least one of a live video and an archive video, audio information corresponding to the video, and distribution destination information for distributing the video and the audio information; discrimination means for discriminating an aspect ratio between the surfaces based on an image of each surface captured by an imaging device installed in the space; image area cut-out means for cut-out each still image constituting the video acquired by the video acquisition means into a plurality of image areas to be assigned to each surface based on the positional relationship between the surfaces, the distribution destination information, and the aspect ratio between the surfaces discriminated by the discrimination means; The system comprises an extraction means for extracting audience information consisting of the characteristics of each image area outputted, and one or more of the audience's position in the space, line of sight, head direction, and sound emitted by the audience; an allocation means for assigning each of the image areas to each of the surfaces and assigning the sound information based on each of the assigned image areas; and a data transmission means for transmitting data including each of the image areas assigned by the allocation means or at least the sound information to each playback device including at least one of each display device for playing back the image areas on each of the surfaces, or each audio device for playing back the sound information, via mutually different channels, and the video acquisition means is characterized in that it re-sets the shooting conditions for the live video based on the characteristics of each image area extracted by the extraction means and the audience information.

[0016] The image area display device of the eighth invention is characterized in that, in an image area display space in which an image area is played back on each of the rectangular surfaces surrounding the space, the image acquisition means acquires each of the rectangular surfaces surrounding the space, a moving image, and distribution destination information to which the moving image is delivered, the image area display device comprises: a discrimination means for determining the aspect ratio between the surfaces based on an image of each surface captured by an imaging device installed within the space, an image area cut-out means for cut-out a still image to be displayed into a plurality of image areas to be allocated to each surface based on the aspect ratio between the surfaces determined by the discrimination means and the distribution destination information acquired by the image acquisition means, and an allocation means for allocating each of the image areas cut out by the image area cut-out means to each of the surfaces.

[0017] The image area generation program of the 9th invention is an image area generation program for generating an image area to be displayed on each rectangular surface surrounding a space, characterized in that it comprises a moving image acquisition step for acquiring a moving image and distribution destination information to which the moving image is to be distributed, a determination step for determining an aspect ratio between the surfaces based on an image of each surface captured by an imaging device installed in the space, an image area cut-out step for cut-out a still image to be displayed into a plurality of image areas to be assigned to each surface based on the aspect ratio between the surfaces determined by the determination step and the distribution destination information acquired by the moving image acquisition step, and an allocation step for allocating each image area cut out by the image area cut-out step to each surface. Effect of the Invention

[0018] According to the present invention having the above-mentioned configuration, a spectator can view the image area displayed on each surface by entering the space. Since this image area is originally an omnidirectional video cut into six surfaces, the spectator in this space can enjoy the feeling of standing in the center of the omnidirectional video by viewing the image area displayed on each surface. When the spectator views each surface, the image area displayed on the viewed surface comes into view. That is, the spectator can obtain a sensation similar to that of VR because the image area corresponding to the viewed direction comes into view. Moreover, the spectator can experience a sense of realism as if he or she were actually in that place in the space without wearing a head-mounted image display device of the glasses type or goggle type required for experiencing VR. Therefore, it is possible to eliminate the feeling of pressure and inconvenience associated with wearing a head-mounted image display device, and the trouble of wearing it. In addition, there is no effect on the body, such as so-called VR sickness, which is caused by a discrepancy between the information obtained visually through the head-mounted image display device and the information received by the real body.

[0019] Furthermore, according to the present invention, multiple spectators can enter the space at the same time and view a common image area, making it possible for multiple people to simultaneously share omnidirectional images within a single virtual space, something that was not possible with conventional VR.

[0020] According to the present invention, each image area can be transmitted to a display device independently via a different communication path, and content can be provided to a space at high speed and low cost. Moreover, time-series inconsistencies between each image area that may occur when each image area is transmitted independently via a different communication path can be eliminated through a synchronization adjustment process.

[0021] Furthermore, according to the present invention, at least one of a live video and an archive video, audio information corresponding to the video, and destination information for distributing the video and audio information are acquired. Therefore, based on the destination information, it is possible to determine the characteristics of each image area cut into a plurality of image areas and the characteristics of the space, and assign each surface and audio information. This makes it possible to transmit data including at least one of each image area and audio information to each reproduction device for reproducing the image area and audio information on each surface through different channels, and it is possible to realize simultaneous sharing of video images and audio information by multiple people in an omnidirectional video in one virtual space.

[0022] Furthermore, according to the present invention, spectator information consisting of the characteristics of each image area and one or more of the spectator's position in the space, line of sight, head direction, and sound emitted by the spectator is extracted. Therefore, audio information can be interactively assigned to each surface based on the state of the spectator relative to the live video. This makes it possible to transmit data including at least each image area and audio information to each playback device for playing back the image area and audio information on each surface through different channels, thereby realizing simultaneous sharing of video images and audio information by multiple people in an omnidirectional video space. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an image area generating system to which the present invention is applied. [Diagram 2] FIG. 2 is a perspective view of a space surrounded by six rectangular faces. [Diagram 3] FIG. 3 is a diagram showing an example in which a plurality of display devices project and display images onto a common surface. [Figure 4] FIG. 4 is a detailed block diagram of the control device. [Diagram 5] FIG. 5 is a flow diagram showing each operation of the image region generating system. [Figure 6] FIG. 6 is a diagram showing an example in which one of the still images constituting the omnidirectional video is illustrated on a rectangular plane. [Figure 7] FIG. 7 is a diagram showing an example of each spherical image region constituting an omnidirectional video captured by an omnidirectional imaging device. [Figure 8] FIG. 8 is a diagram showing an example of dividing still images constituting an omnidirectional video into a plurality of image regions. [Figure 9] FIG. 9 is a diagram showing an example in which the image area allocated to each surface is adjusted to have a rectangular shape. [Figure 10] FIG. 10 is a diagram showing an image in which data for each image region is transmitted to a display device in a time-series manner. [Figure 11] FIG. 11 is a diagram showing an example in which each image area cut out from an omnidirectional video is displayed on each surface via each display device. [Figure 12] FIG. 12 is a diagram showing an example of a method for extracting an image area when the display device is a projection display device that projects and displays an image area on a surface. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] 1 shows an overall configuration diagram of an image area generating system 1 to which the present invention is applied. This image area generating system 1 is centered around a control device 2, and includes a recording module 3 connected to the control device 2, a display device 7 that displays video and an audio device 8 that plays audio as playback devices connected from the control device 2 via a communication network 5, and a video storage unit 9 that stores various video contents, audio files, and the like. This image area generating system 1 may further include a space 6 in which the display device 7 and audio device 8 are provided. The space 6 may be installed, for example, at multiple locations, each of which may be installed individually or in cooperation with each other.

[0025] The control device 2 serves as a so-called central control device that controls the entire image area generating system 1. The control device 2 is embodied as, for example, a personal computer (PC), but is not limited thereto, and may be embodied as a server or a dedicated device, or may be embodied as a mobile information terminal, a tablet terminal, or the like.

[0026] The recording module 3 is used to pre-record alternative footage based on past events separate from the real-life event, and includes an omni-directional imaging device 31 and a microphone 32 .

[0027] The omnidirectional imaging device 31 is configured to be capable of simultaneously capturing images in all directions (360° horizontally and 360° vertically) with the imaging device main body at the center. By recording a video using this omnidirectional imaging device 31, videos in all directions (hereinafter referred to as omnidirectional videos) can be captured simultaneously without omission. For this reason, for example, when capturing images of an urban space, if a vehicle or person moves, the moving vehicle or person can be recorded in all directions in time series as moving images. In addition, the omnidirectional imaging device 31 may be fixed in one place to capture omnidirectional videos continuously, or the omnidirectional imaging device 31 itself may be mounted on a moving object such as an unmanned aerial vehicle, a vehicle, or a helicopter to continuously record omnidirectional videos.

[0028] This makes it possible to obtain a moving image in which the user is viewing in all directions as if he or she were riding on such a moving object. The omnidirectional video captured by the omnidirectional imaging device 31 is output to the control device 2. Note that the omnidirectional imaging device 31 may not only be directly connected to the control device 2, but may also be connected via a communication network (not shown) that is constituted by the Internet, a LAN (Local Area Network), or the like.

[0029] The microphone 32 collects surrounding sounds and converts them into audio signals. The microphone 32 transmits the converted audio signals to the control device 2 via an interface. The microphone 32 is necessary to realize live video playback, but is not a particularly essential component and may be omitted.

[0030] The communication network 5 is an internet network or the like that connects the control device 2, the display device 7, and the video device 8 via communication lines. When the recording module 3, the control device 2, the display device 7, and the audio device 8 are operated within a certain small area, the communication network 5 may be configured as a LAN. The communication network 5 is not limited to a wired communication network, and may be realized as a wireless communication network.

[0031] As shown in FIG. 2, the space 6 is a space surrounded by six rectangular faces 61a to 61f. The space 6 is composed of four wall faces and faces 61a to 61f corresponding to a ceiling and a floor, for example, like a room. In this case, the space 6 may be provided with a door (not shown) so that people can enter and exit the space. The space 6 is not limited to a completely closed space surrounded by the six faces 61a to 61f, but may be an open space in which one or more faces 61 are omitted, or an open space in which only a part of one or more faces 6 is opened. The inside of the space 6 may have various structures such as various shapes, irregularities, protrusions, fixtures, etc. other than the faces 61a to 61f.

[0032] The image area generating system 1 reproduces the generated moving image and audio information corresponding to the moving image by a reproduction device. The reproduction device is composed of, for example, a display device 7 and an audio device 8. The display device 7 is composed of a projection display device that projects and displays the image area on a surface, such as a so-called projector. This display device 7 is not limited to being composed of a projection display device, and may be composed of a display for displaying the image area on a surface, such as a liquid crystal display, an organic EL display, or even an LED display.

[0033] When the display device 7 includes, for example, a speaker that outputs voice, music, etc., it may function as a device that reproduces sound. The display device 7 may include various speakers, and may be linked to, for example, an audio device 8 separate from the display device 7. For example, when recording and reproducing sound, the audio device 8 reproduces the sound based on the three-dimensional direction, distance, spread, etc. of the sound.

[0034] The acoustic device 8 reproduces realistic, three-dimensional sound, for example, by using a number of elements that make up the sound. The multiple elements include, for example, a "volume difference" that reproduces the sound source's sound image localization by the attenuation of the sound volume due to the distance between the acoustic device 8 and the target (target person) or in space, or the difference in intensity between the two ears, a "time difference" that reproduces the sound source's sound image localization by the time difference when the sound wave reaches the target, a "change in frequency characteristics" that reproduces the sound source's sound image localization by the change in frequency characteristics due to the transmission or obstruction of sound waves, a "phase change" that reproduces the sound source's sound image localization by the change in phase due to the transmission or obstruction of sound waves, and a "change in reverberation" that reproduces the sound field of the surrounding environment by the reverberation characteristics.

[0035] The audio device 8 performs a rendering process of stereophonic sound (3D sound) for controlling a sound field in the three-dimensional space of the space 6, based on a plurality of elements, such as the type of video, e.g., live video or archive video, shooting information, e.g., the characteristics of each image area, and audience information, such as the position of the audience M in the space, line of sight, head direction, and voice emitted by the audience M. In the rendering process of stereophonic sound, the audio device 8 performs processing, for example, by a known feature prediction method and ray tracing method, using various processes and technologies, such as known "feature prediction technology," "sound ray method / geometric acoustic modeling technology," and "adaptive rectangular decomposition."

[0036] The display device 7 and the audio device 8 function as playback devices that play back moving images and audio information, respectively. The control device 2 displays the image area generated by the control device 2 on each of the surfaces 61a to 61f that form the space 6 as shown in FIG. 2 via the display device 7. In the example of FIG. 2, the display devices 7a to 7f are configured as projection display devices, for example, as projectors, and the display device 7g is configured as an LED display. The audio device 8 is configured as a plurality of speaker units that play 3D audio and stereophonic audio, and is installed behind each surface of the space 6 (for example, the back of the surface 61b, etc.). If the display devices 7a to 7f are equipped with speakers, the audio device 8 may be configured to play back together with the speakers.

[0037] The display device 7a is attached near the upper end of the surface 61a, and projects and displays an image on the surface 61c facing the surface 61a. The display device 7b is attached near the upper end of the surface 61b, and projects and displays an image on the surface 61d facing the surface 61b. The display device 7c is attached to the middle of the surface 61b, and the display device 7e is attached to the middle of the surface 61d, and projects and displays an image on the common surface 61e. The display device 7d is attached near the upper end of the surface 61d, and projects and displays an image on the surface 61b facing the surface 61d. The display device 7f is attached near the upper end of the surface 61c, and projects and displays an image on the surface 61a facing the surface 61c. The display device 7g, which is an LED display, displays an image on the surface 61f.

[0038] In addition, the combination of the display device 7 on which image is displayed on which surface 61 constituting the space 6 includes any combination other than the above example. For each surface 61, an image may be displayed not only by one display device 7, but also by a combination of multiple display devices 7. The example of FIG. 3 shows an example in which an image is projected and displayed on a common surface 61e by the display device 7c and the display device 7e. That is, the area of ​​one surface 61e is divided in half, an image is projected and displayed on one area by the display device 7c, and an image is projected and displayed on the other area by the display device 7e. The other surface 61 may also be divided into areas in the same way, and images may be projected and displayed by distributing them to multiple display devices 7. In this case, the audio device 8 may determine the area of ​​each surface divided into halves, and reproduce audio information according to the images distributed and displayed on each area.

[0039] The audio device 8 also reproduces audio information corresponding to the acquired video. For example, the audio device 8 may reproduce audio information of the determined characteristics according to the determination result of the characteristics of each cut-out image region. In this way, the audio device 8 can reproduce data including audio information assigned to each reproduction device and transmitted through different channels as 3D audio.

[0040] For example, one audio device 8 is attached to the back surface of surface 61b, and reproduces sound for space 6 surrounded by six rectangular surfaces 61a to 61f as 3D sound. The audio device 8 may be configured such that a plurality of audio devices 8 are installed on the six surfaces 61a to 61f (not shown). This makes it possible to reproduce the direction, distance, spread, etc. of three-dimensional sound in accordance with the moving images displayed in space 6 surrounded by the six surfaces 61a to 61f.

[0041] The video storage unit 9 is a database for storing at least one of live video and archive video to be displayed on the display device 7, and audio information associated with these video. The video storage unit 9 stores in advance omnidirectional video including audio information already captured by an imaging device (not shown) other than the above-mentioned omnidirectional imaging device 31. Various types of video including audio information stored in the video storage unit 9 may be not only the above-mentioned omnidirectional video and audio information, but also ordinary two-dimensional video and audio information. The omnidirectional video stored in the video storage unit 9 is sent to the control device 2 via the communication network 5.

[0042] Next, a detailed block configuration of the control device 2 will be described. As shown in Fig. 4, the control device 2 includes a first moving image acquisition unit 21, a second moving image acquisition unit 23, a spatial information acquisition unit 26, an audio data acquisition unit 35, and an operation unit 25, and further includes a control unit 28 to which the first moving image acquisition unit 21, the second moving image acquisition unit 23, the spatial information acquisition unit 26, the audio data acquisition unit 35, and the operation unit 25 are connected. Furthermore, I / Fs (interfaces) 29-1, 29-2, 92-3, ... 29-n for transmitting data of each image region P1, P2, P3, ... Pn to be output are connected to the control unit 28. Furthermore, an I / F 30-1 for transmitting data of the audio information S1 to be output is connected to the control unit 28. Note that the audio information S1 may be configured to be connected to, for example, a plurality of display devices 7a, ..., ... 7n and a plurality of audio devices 8 (not shown).

[0043] The control device 2 is composed of a PC or the like, and in addition to these components, it also has a CPU (Central Processing Unit) as a so-called central computing unit for controlling each component, a ROM (Read Only Memory) in which programs for controlling the hardware resources of the entire control device 2 are stored, a RAM (Random Access Memory) used as a working area for storing and expanding data, and an image processing unit for performing various image processing on the omnidirectional video and for cutting out each image area P1 to Pn.

[0044] The first video acquisition unit 21 acquires the omnidirectional video stored in the video storage unit 9 via the communication network 5. The first video acquisition unit 21 may acquire, for example, video stored in the video storage unit 9 as archived video. The archived video may be, for example, past video stored in each video server as a known video providing service on the Web or cloud. The video includes, for example, sound, background at the time of shooting, surrounding sounds, music data added by the photographer, or audio information (2D / 3D audio information, sound source information, audio equipment information, sound effect information, setting values, parameters, etc.), and various music data and information may be associated with the audio information individually or commonly.

[0045] The second video acquisition unit 23 acquires omnidirectional video captured by the omnidirectional imaging device 31. The second video acquisition unit 23 may acquire omnidirectional video of various locations in real time as live video by, for example, the omnidirectional imaging device 31 (for example, a fixed camera, a stationary camera, etc.) installed in various locations. The second video acquisition unit 23 may acquire video of the inside of the space 6 (such as video of people inside and motion sensor information showing the positions and movements of parts of people) by, for example, the omnidirectional imaging device 31 installed in the space 6, or other imaging devices or sensors, or sensors held by spectators (not shown).

[0046] Multiple omnidirectional imaging devices 31 may be installed in multiple spaces 6, for example, and various information such as the position, line of sight, head direction, and voices emitted by the spectators of the people inside (spectators M) may be obtained individually or together as spectator information.

[0047] Second video acquisition unit 23 may also acquire various information and data such as location information of the location where omnidirectional imaging device 31 is installed, surrounding environmental information, imaging date and time, weather, etc. The video acquired by second video acquisition unit 23 may be stored in video storage unit 9 via communication network 5 by control unit 28, for example.

[0048] The spatial information acquisition unit 26 acquires the space 6 in which the image regions P1 to Pn are actually displayed and various information related to the space 6. The spatial information acquisition unit 26 acquires various information related to the shape of the space 6, such as the aspect ratio between the surfaces 61 of the space 6. The information acquired by the spatial information acquisition unit 26 includes information related to the arrangement of the display devices 7 installed on the surfaces 61 of the space 6, and information on which display device 7 will display an image on each surface 61, not only when an image is displayed on each surface 61 by one display device 7 as described above, but also when a plurality of display devices 7 are combined and displayed on each surface 61, information on the allocation of the display devices.

[0049] Furthermore, the spatial information acquisition unit 26 acquires various information related to the space 6 in which the acoustic information S1 is actually played. The spatial information acquisition unit 26 acquires various information related to the shape, material, and echogenic objects of the space 6, such as the aspect ratio between the sides 61 of the space 6. The information acquired by the spatial information acquisition unit 26 includes information on the arrangement of the acoustic devices 8 installed on the sides 61 of the space 6, and not only when acoustic information is played by one acoustic device 8 for each side 61 as described above, but also information on which acoustic device 8 will play acoustic information to the audience in the space 6, individual acoustic modules (acoustic units) constituting the acoustic device 8, and when playing the acoustic information in combination with a plurality of display devices 7 for each side 61, various information on the allocation, timing, directivity, and the like.

[0050] When the display device 7 is configured as a playback device in combination with a projection display device or an audio device 8, the spatial information acquisition unit 26 may also acquire information on the relative positions of the display device 7, or the projection direction and angle of view of each projection display device or playback device relative to the surface 61. The spatial information acquisition unit 26 transmits the acquired information on the space 6 to the control unit 28.

[0051] Furthermore, when configured as a playback device in combination with one or more audio devices 8 and display devices 7, the spatial information acquisition unit 26 may also acquire information such as their layout relationship, or the directivity of the audio device 8 with respect to the space 6, surface 61, and audience, and the intensity of 3D sound. The spatial information acquisition unit 26 transmits the acquired information on the space 6, surface 61, audience, etc. to the control unit 28. As information on the audience, etc., predetermined, assumed audience information may be transmitted to the control unit 28.

[0052] The audio data acquisition unit 35 acquires audio from the microphone 32 or the like and stores it. The audio may be acquired from the microphone 32, for example, via a public communication network via wired or wireless connection, or audio data recorded on a recording medium may be read and recorded. The audio data acquisition unit 35 may acquire, in addition to audio, various types of music or background music, or sound information or acoustic data at the location where the microphone 32 is installed. The audio data acquisition unit 35 may acquire audio data from multiple sound sources via multiple microphones 32 or the like.

[0053] The audio data acquisition unit 35 may acquire video images (videos of people inside and motion sensor information showing the positions and movements of parts of people) inside the space 6, for example, using a microphone provided on an omnidirectional imaging device 31 installed in the space 6, or another microphone 32, or a microphone held by a spectator (not shown). For example, multiple omnidirectional imaging devices 31 may be installed in multiple spaces 6, and various information such as the positions, line of sight, head direction, and sounds emitted by the spectators inside the space 6 may be acquired individually or together as spectator information.

[0054] The operation unit 25 is realized by a keyboard and a touch panel, and an execution command for executing a program is input by the user. When the execution command is input by the user, the operation unit 25 notifies the control unit 28. The control unit 28, which receives the notification, executes the desired processing operation in cooperation with each component, including the determination unit 27.

[0055] Control unit 28 is a so-called central control unit for controlling each component implemented in control device 2 by transmitting a control signal via an internal bus. Control unit 28 also transmits various control commands via the internal bus in response to operations via operation unit 25. Control unit 28 accepts input of various data of moving images and audio information from first moving image acquisition unit 21 and second moving image acquisition unit 23, respectively.

[0056] The control unit 28 cuts out a plurality of image regions P1, P2, . . . , Pn for each still image constituting the input-accepted moving image, as described later. The data of the moving image including the cut-out image regions P1, P2, . . . , Pn is transmitted on different channels via the I / Fs 29-1, 29-2, . . . , 29-n, respectively. The control unit 28 cuts out a plurality of image regions P1, P2, . . . , Pn for each still image constituting the input-accepted moving image, as described later. The data of the moving image including the cut-out image regions P1, P2, . . . , Pn is transmitted on different channels via the I / Fs 29-1, 29-2, . . . , 29-n, respectively. Furthermore, the control unit 28 transmits, for each piece of acoustic data constituting the input-accepted acoustic information, acoustic information S1 on a channel different from that of the data of the different moving images via the I / F 30-1, as described later.

[0057] Each of the I / Fs 29-1, 29-2, ..., 29-n, and I / F 30-1 serves as an interface for establishing a communication link between the control device 2 and the display device 7 and audio device 8 as a playback device. The I / Fs 29-1, 29-2, ..., 29-n are not limited to being provided individually for the multiple image areas P1, P2, ..., Pn cut out by the control unit 28, and the I / F 30-1 is not limited to being provided individually for the multiple pieces of audio information S1 cut out by the control unit 28, and may be configured as a common interface unit.

[0058] Next, the operation of the image area generating system 1 to which the present invention is applied and which has the above-mentioned configuration will be described.

[0059] 5 is a flow diagram showing each operation of the image area generating system 1. First, in step S11, the control device 2 acquires a moving image. The control device 2 acquires the moving image via the first moving image acquisition unit 21 and the second moving image acquisition unit 23 described above. That is, when an omnidirectional video stored in the moving image storage unit 9 is sent via the communication network 5, it is acquired via the first moving image acquisition unit 21. Also, when an omnidirectional video is captured via the omnidirectional imaging device 31, it is acquired via the second moving image acquisition unit 23.

[0060] In step S11, the control device 2 acquires audio information and destination information related to the omnidirectional video in addition to the omnidirectional video. For example, the control device 2 may acquire audio information related to the video and destination information to which the video and audio information are delivered individually or collectively by including them in the omnidirectional video.

[0061] The control device 2 acquires the omnidirectional video (moving image), audio information, and distribution destination information via the above-mentioned first moving image acquisition unit 21 and second moving image acquisition unit 23. For example, when an omnidirectional video stored in the moving image storage unit 9 is sent via the communication network 5, the control device 2 acquires it via the first moving image acquisition unit 21, and when an omnidirectional video is captured via the omnidirectional imaging device 31, the control device 2 acquires it via the second moving image acquisition unit 23.

[0062] When the omnidirectional video and the omnidirectional video acquired by the first video acquisition unit 21 and the second video acquisition unit 23 include various information such as audio information and distribution destination information, these pieces of information may also be sent to the control unit 28. The distribution destination information includes, for example, various information related to the distribution of the omnidirectional video. The distribution destination information may include, for example, contract information (distribution conditions, billing information, point information, etc.), facility information related to the distribution (space 6 information, projection facility information, audio facility information, lighting information, etc.), customer information related to the audience to whom the distribution can be made (member information, gender, age, height, hobby information, group information, etc., and motion information indicating the line of sight, posture, movement, etc. of the customer who acquired the omnidirectional video in the space 6 in real time, etc.

[0063] The control unit 28 cuts out image areas, which will be described below, for each still image constituting the omnidirectional video sent from the first video acquisition unit 21 and the second video acquisition unit 23 (step S12).

[0064] Fig. 6 shows one of the still images constituting the omnidirectional video on a rectangular plane. Each still image constituting the omnidirectional video captured by the omnidirectional imaging device 31 can be divided into spherical image areas Q1-a, Q1-b, Q2-a, Q2-b, Q3-a, Q3-b, Q4-a, Q4-b, Q5, and Q6 that form a sphere as a whole, as shown in Fig. 7. The spherical image areas Q1-a, Q1-b, Q2-a, Q2-b, Q3-a, Q3-b, Q4-a, Q4-b, Q5, and Q6 are redrawn in a plane to become the still images constituting the omnidirectional video shown in Fig. 6.

[0065] The control unit 28 cuts out image areas P1, P2, ..., Pn in the figure for each still image constituting such an omnidirectional video. In the example of Fig. 6, six image areas P1 to P6 are cut out. For example, when cutting out the image areas P1, P2, ..., Pn, the control unit 28 may cut out the distribution destination information, various information contained in the audio information, and each still image constituting the video into a plurality of image areas according to the arrangement relationship of each surface.

[0066] The control unit 28 further includes, for example, an extraction unit. The extraction unit, for example, determines the characteristics of the moving image displayed from each cut-out image area, and extracts audience information consisting of the characteristics of each determined image area and one or more of the real-time position of the audience in the space 6, the line of sight, the direction of the head, and the sound emitted by the audience. The extraction unit acquires, for example, motion information indicating the characteristics of various information (audience information) such as the position of the audience in the space 6, the line of sight, the direction of the head, and the sound emitted by the audience M, using the second moving image acquisition unit 23 or other known sensors, and specifies the information by processing such as image discrimination and sound discrimination, and extracts motion information indicating the real-time line of sight, posture, and movement of each audience member. The extraction unit, for example, associates each image area (still image or moving image) in the space 6 with the real-time motion information of the audience and stores it in the moving image storage unit 9.

[0067] Simultaneously with the cut-out process of the image regions P1 to P6, or after the cut-out process is completed, the cut-out image regions P1 to P6 are assigned to the respective faces 61a to 61f (step S13). Image region P1 is assigned to face 61a shown in Fig. 2, image region P2 is assigned to face 61b, image region P3 is assigned to face 61c, image region P4 is assigned to face 61d, image region P5 is assigned to face 61e, and image region P6 is assigned to face 61f. That is, in this cut-out example, an image region P is assigned to each face 61. In the case where a combination of multiple image regions P is to be displayed on one face 61, the multiple image regions P to be displayed are similarly assigned to the one face 61.

[0068] Furthermore, at the same time as allocating the image regions P1 to P6 to the surfaces 61a to 61f, the audio information S1 is also allocated to the audio device 8 shown in Fig. 2. For example, if speakers of the display devices 7a to 7n and other audio playback devices (not shown) are set in addition to the audio device 8, the control unit 28 divides the audio information S1 in accordance with the cut-out process of the image regions P1 to P6, and allocates it to each of the multiple audio devices.

[0069] Furthermore, the control unit 28 may assign each image area to each surface based on, for example, the characteristics of each extracted image area and audio information, and the characteristics of the space 6 etc. (size, material, number of spectators, characteristics, etc.), and may further assign the entire audio information to be played in the space 6 (for a large group) or a portion of it (for specific people, children, adults, by fee, etc.) to the entire space 6 or to each surface individually.

[0070] The control unit 28 may cut out each still image constituting the moving image into a plurality of image regions according to the arrangement of the surfaces, for example, according to the length and effect of the acquired acoustic information, and assign each of the cut-out image regions P1 to P6 to each of the surfaces 61a to 61f. Furthermore, the control unit 28 may determine the directivity of the acquired acoustic information for each of the assigned surfaces 61a to 61f, and assign the playback timing, playback pattern, sound effects, and the like of the acoustic information so that it can be played back in the space 6. This makes it possible to reliably play back the acoustic information to the audience M in the space 6 with pinpoint accuracy along with the moving image.

[0071] Furthermore, the control unit 28 may cut out each still image constituting a new moving image into a plurality of image areas according to the arrangement of each surface, based on, for example, the characteristics of each image area extracted by the extraction unit, and audience information (motion information) consisting of one or more of the audience's positions in the space 6, their gazes, their head directions, and their sounds, and assign each of the cut-out image areas P1 to P6 to each of the surfaces 61a to 61f. This makes it possible to interactively assign omnidirectional images and sound information to each surface based on the audience's movements in response to the live video distributed in the space 6.

[0072] The constituent still images that make up the omnidirectional video are thus divided into a plurality of image regions P without leaving any remainder. The shape of the boundary between these image regions P is determined based on the aspect ratio between the surfaces 61. Similarly, the sound information corresponding to the still images is also assigned based on the divided plurality of image regions P.

[0073] For example, it is assumed that the boundaries of image regions P1 to P6 shown in Fig. 8(a) correspond to the aspect ratio of faces 61a to 61f of a certain space 6. In this case, if the area of ​​the other space 6 is smaller than that of the certain space 6 and the aspect ratio of faces 61a to 61d is also different from that of the certain space 6, then, for example, as shown in Fig. 8(b), the boundaries of image regions P1 to P4 are expanded in the vertical direction, and the boundaries of image regions P5 and P6 are adjusted to have a shape compressed in the vertical direction.

[0074] The acoustic information is adjusted as acoustic information S1 to be sent to the space 6 according to the shape and movement of the adjusted image area, for example, and is reproduced by the acoustic device 8. Note that, if each of the display devices 7a-7n is equipped with a speaker or the like, the acoustic information may be reproduced from the speaker. Furthermore, if the space 6 is equipped with a plurality of the acoustic device 8, the speakers of each of the display devices 7a-7n, and other acoustic reproduction devices (not shown), the control unit 8 controls the acoustic information for each of the surfaces 61a-61f and reproduces it.

[0075] In step S13, the image regions P1 to P6 assigned to each surface 61 may be adjusted so that each of the image regions P1 to P6 has a rectangular shape. In this case, in the case of an image region P2 as shown in Fig. 9(a), for example, image processing is performed to extend the upper and lower ends in the direction of the arrow toward the dotted line in the figure, thereby obtaining an image region P2 processed into a rectangular shape as shown in Fig. 9(b).

[0076] Next, the process proceeds to step S14, where the control unit 28 transmits the image areas P1-Pn thus generated via the I / F 29 and the audio information S1 on different channels. The term "channel" here refers to a communication line. In other words, transmitting on different channels means that the data for the image areas P1-Pn and the audio information S1 are transmitted separately via different communication lines. Then, the data for the image areas P1-Pn and the audio information S1 thus separated for each communication line are transmitted as they are to the playback devices, which are the display device 7 and the audio device 8 (or the individual audio modules that make up the audio device 8), respectively, independently of each other.

[0077] In the example of Fig. 4, image area P1 is independently transmitted to display device 7a, image area P2 is independently transmitted to display device 7b, image area P3 is independently transmitted to display device 7c, and image area Pn is independently transmitted to display device 7n. During this time, the data of each of image areas P1 to Pn is not collected in one place, but is transmitted to display device 7 via independent communication paths. Furthermore, acoustic information S1 is independently transmitted to audio device 8. Note that each of image areas P1 to Pn independently transmitted to each of display devices 7a to 7n may be transmitted including, for example, acoustic information S1 or individually subdivided acoustic information constituting acoustic information S1.

[0078] The original omnidirectional video is composed of a large number of still images over one second at a preset frame rate (24 fps, 30 fps, 60 fps, etc.). Continuously transmitting the image areas P1 to Pn into which such a large number of still images are divided in a time series requires a considerable amount of communication traffic. If the continuous transmission of such image areas P1 to Pn were to be performed through one communication path, a considerable communication time would be required and the communication cost would be excessive. For this reason, in the present invention, the image areas P1 to Pn are transmitted to the display device 7 and the audio device 8 that transmits the audio information S1, or to a playback device composed of the display device 7 and the audio device 8, through different communication paths, thereby reducing the transmission rate of the image area P in each communication path, and as a result, the data of the image area P can be transmitted to the display device 7 at high speed and at low cost.

[0079] The image areas P1-Pn and the audio information S1 may be transmitted to the display device 7 and the audio device 8 or the playback device using different frequency channels in addition to different communication paths. By using different frequency channels for the image areas P1-Pn to the display device 7 and the audio device 8 or the playback device, it becomes possible to transmit the image areas P1-Pn with high communication quality without interfering with each other.

[0080] Next, the process proceeds to step S15, where adjustment is performed to synchronize the image areas of the data to be transmitted to the display device 7 and the audio device 8 in time series with each other.

[0081] 10 shows an image in which data for each image region P1 to Pn is transmitted continuously in time series to display device 7. Data for each image region P1 to Pn extracted from a still image constituting the omnidirectional video is transmitted sequentially to display device 7. Image regions P1 to Pn are similarly extracted from the next still image constituting the omnidirectional video and transmitted to display device 7. When this process is repeated, the result is a form in which each image region P1 to Pn is transmitted from the beginning of a frame along the axis of time t, just as shown in FIG.

[0082] Time series identification information may be sequentially assigned to the data stream of each image region P1-Pn transmitted along the axis of time t in this manner. This time series identification information may be something like a so-called time stamp, and may be assigned in correspondence with the time at which the image regions P1-Pn are generated. The time series identification information may also correspond to frame numbers assigned in chronological order to still images constituting the omnidirectional video. In other words, the image regions P1-Pn cut out from the frames of the same still image may be assigned time series identification information corresponding to the same frame number.

[0083] As a result, the image regions P1 to Pn to which time-series identification information corresponding to the same frame number is assigned are synchronized with one another in time series, and it becomes possible to display the image regions P1 to Pn without any misalignment with one another.

[0084] 10, for the sake of simplicity, "#" in Pn-# is assumed to be time-series identification information. This time-series identification information is assigned in the order of 1, 2, 3, . . ., #, . . . from oldest to newest.

[0085] As shown in FIG. 10, for example, in order to synchronize image areas P1, P3, and P4, the time series identification information is identified between the image areas P1-1, P3-1, and P4-1 that are sent first, to check whether they are chronologically consistent with each other.

[0086] For example, if the time-series identification information corresponds to frame numbers sequentially assigned to still images constituting the omnidirectional video, and the time-series identification information corresponding to the frame numbers is common, it can be determined that the images are synchronized in time series. Also, if the time-series identification information corresponds to the time at which the image regions P1 to Pn are generated, and it can be assumed that the generation times of the image regions P1 to Pn are always the same without any discrepancy, it can be determined that the images are synchronized in time series if the time-series identification information is common.

[0087] As a result of identifying the time series identification information added to the end of the image areas P1-1, P3-1, and P4-1 sent first, it can be determined that the time series identification information is identical to each other and therefore that they are synchronized chronologically. At the next timing, the time series identification information is common to the image areas P3-2 and P4-2, but the image area P1 is missing. In such a case, it can be determined that the image areas P are not synchronized with each other. Furthermore, at the next timing, if the time series identification information added to the end of the image area P1-2 and P4-4 does not match that of the image area P3-3, it can also be determined that the image areas P are not synchronized with each other.

[0088] In this way, when it is determined that the image regions are not synchronized with each other as a result of the determination through the time series identification information, adjustment is performed between each of the image regions P1 to Pn to synchronize them with each other in time series. For example, as described above, the image regions P3-2 and P4-2 have the same time series identification information, but the image region P1 is missing at the same timing, and the image region P1-2 is linked to the image region P3-3 at a later timing. In this case, the image regions P1-2 and P4-2 are linked to the image regions P3-2 and P4-2 whose time series identification information is consistent with the image region P1-2, and are synchronized. Alternatively, when the image region P1-2 itself, which has the same timing as the image regions P3-2 and P4-2, is completely missing, the image region P1-2 itself may be newly generated. In such a case, the missing image region P1-2 may be generated by interpolating pixels based on the image regions P1-1 and P1-3 before and after it using a known technique, or one of the image regions P1-1 and P1-3 before and after it may be inserted as it is.

[0089] The adjustment for achieving synchronization using such time series identification information in step S15 may be performed via a server (not shown) provided in the communication network 5, or may be performed between the display devices 7 that actually receive the data of these image areas P1 to Pn. When performing adjustment for achieving synchronization between the display devices 7, the adjustment may be realized by the display devices 7 communicating with each other. Also, the adjustment for achieving synchronization using the time series identification information may be performed within the control device 2. In either case, since the data of the image areas P1 to Pn are transmitted via different channels, the adjustment for synchronization is performed within the control device 2 before transmission, between the display devices 7 after transmission, or within the communication network 5.

[0090] In step S15, sound information S1 is sent to the sound device 8 in conjunction with the adjusted data of each image region P1 to Pn and the transmission of each data to the display device 7. The sound information S1 is cut out for each time and for each effect according to, for example, a certain time period, spatial characteristics, sound type, audience information, etc., and sent to the sound device 8. By repeatedly executing these steps for the sound information S1 as well, the sound information S1 is transmitted in accordance with the beginning of the frame of each image region P1 to Pn on the axis of time t, as shown in Fig. 10.

[0091] The adjustment for achieving synchronization using such time series identification information for each image area P1 to Pn and audio information S1 may be performed via a server (not shown) provided in communication network 5, or may be performed by the display device 7 and audio device 8, or a playback device, which actually receive the data for each of these image areas P1 to Pn and audio information S1.

[0092] In this way, the data of the image area P and the audio information S1, which are synchronized with each other in time series, are sent to each display device 7 that displays the assigned surface 61, and to the audio device 8, or to a playback device.

[0093] Each display device 7 displays the image area P on each surface 61 (step S16). It has already been determined which display device 7a-7g is used to display the image on each surface 61a-61f. Therefore, the image areas P1-Pn assigned to each surface 61a-61f are transmitted to the display device 7a-7g that displays the surface 61 and displayed thereon. As a result, as shown in FIG. 11, each image area P1-Pn cut out from the omnidirectional video is displayed on each surface 61a-61f via each display device 7a-7g.

[0094] Furthermore, the audio device 8 is installed behind each of the surfaces 61a-61f of the display devices 7a-7g so as to transmit the audio information S1 to each surface 61 of the space 6. It may be already determined which display devices 7a-7g of the space 6 and each surface 61a-61f to transmit the audio information to via the audio device 8. This allows the audio information suitable for the image regions P1-Pn assigned to the space 6 and each surface 61a-61f to be transmitted to the display devices 7a-7g displaying the surface 61. This allows the audio information synchronized with the space 6 and each surface 61a-61f to be reproduced via the audio device 8 in accordance with the image regions P1-Pn cut out from the omnidirectional video, as shown in FIG.

[0095] When a spectator M enters the space 6, the spectator M can view the image areas P1 to Pn displayed on each of the faces 61a to 61g. Since the image areas P1 to Pn are originally cut out from an omnidirectional video into six faces, the spectator M in the space 6 can enjoy the sensation of standing in the center of the omnidirectional video by viewing the image areas P1 to Pn displayed on each of the faces 61a to 61g. When the spectator M views each of the faces 61a to 61g, the spectator M can see the image area P displayed on the face 61 that he / she views. That is, the spectator M can get the same sensation as VR because the image area P corresponding to the direction that he / she views is seen. Moreover, the spectator M can experience a sense of realism as if he / she were actually in the place in the space 6 without wearing a head-mounted image display device of a glasses type or a goggle type that is necessary when experiencing VR. Therefore, the spectator M can eliminate the sense of oppression, the hassle, and the trouble of wearing a head-mounted image display device. In addition, there will be no adverse effects on the body, such as VR sickness, which arises from a discrepancy between the information obtained visually through a head-mounted image display device and the information received by the real body.

[0096] Furthermore, the audience M in the space 6 can enjoy the sensation of standing in the center of an omnidirectional video with images and sounds (stereophonic, 3D sound) by experiencing the image areas P1 to Pn displayed on each surface 61a to 61g together with the sound information S1. The audience M can experience the image area P and sound information S1 displayed on the surface 61 that he / she viewed by viewing each surface 61a to 61g together with the sound information. That is, the audience M can obtain a sensation in the space 6 similar to a real experience by viewing the image area P and sound information according to the direction of the viewer's vision. Moreover, the audience M can experience a sense of realism as if he / she were actually in the place in the space 6 by the 3D sound from the sound device 8 without wearing a head-mounted image display device of a glasses type or goggle type that is necessary when experiencing VR. Therefore, the sense of pressure, the hassle, and the trouble of wearing a head-mounted image display device can be eliminated. In addition, there is no effect on the body, such as so-called VR sickness, caused by a discrepancy between the information obtained visually through the head-mounted image display device and the information received by the real body.

[0097] Furthermore, according to the present invention, multiple spectators M can enter the space 6 at the same time and view the common image area P and audio information S1, realizing simultaneous sharing of omnidirectional images and audio by multiple people in one virtual space, something that was not possible with conventional VR. Furthermore, for example, the common image area P and audio information S1 can be transmitted to other locations via the communication network 5, so that multiple locations can simultaneously view the common image area P and audio information S1 and enjoy it in each of the spaces 6.

[0098] Furthermore, according to the present invention, each of the image areas P1 to Pn can be transmitted independently to the display devices 7a to 7f via different communication paths, making it possible to provide content at high speed and low cost to the space 6. Moreover, time-series inconsistencies between each of the image areas P1 to Pn, which may occur when each of the image areas P1 to Pn is transmitted independently via different communication paths, can be eliminated through the synchronization adjustment process in step S15.

[0099] The space 6 surrounded by the surfaces 61 varies in shape and size depending on the actual situation at the site, and it becomes necessary to flexibly and freely cut out an optimal image area P according to the space 6. According to the present invention, the image area P to be assigned to each surface 61 can be cut out based on the aspect ratio between the surfaces 61, so that the variety of shapes of the space 6 can be accommodated.

[0100] In such a case, when there is a space 6 in which a new image area P is to be displayed, an imaging device is installed in the space 6. This imaging device may be configured as a so-called omnidirectional imaging device that can simultaneously capture images in all directions (360° horizontally and 360° vertically) without omission, centered on the imaging device main body. Also, a plurality of imaging devices that capture normal planar images may be installed in the space 6, and all of the surfaces 61 may be captured by sharing the images among the plurality of imaging devices.

[0101] Such an imaging device captures an image of each surface 61 of the space 6 to be newly displayed, and a well-known image analysis technique is used to determine the aspect ratio between each surface 61. Based on the aspect ratio between each surface 61 thus determined, the control unit 28 cuts out an image area P to be assigned to each surface 61 in the same manner as described above.

[0102] When the display device 7 is a projection display device that projects and displays the image area P on the surface 61, the image area may be cut out by the following method.

[0103] FIG. 12(a) is a side cross-sectional view of a space 6, and FIG. 12(b) is a plan view thereof. Display devices 7m, 7w, each of which is a projection display device, and an audio device 8 are provided on surfaces 61b, 61d, which constitute the side surfaces of the space 6. The display devices 7m, 7n project and display an image area P toward the surface 61e, which constitutes the ceiling, and obtain the projection direction and the angle of view θ of the display device 7 at that time. The display device 7w is provided on the surface 61e, which constitutes the ceiling, and projects and displays an image area P in four directions toward the four surfaces 61a to 61d, which constitute the side surfaces, and obtains the projection direction and the angle of view φ at that time. Information regarding the positional relationship between the display devices 7m, 7n, 7w, and the audio device 8 is also obtained.

[0104] The projection direction, angle of view, and positional relationship may be acquired not only by input via the operation unit 25, but also automatically through imaging by an imaging device installed in the space 6 as described above.

[0105] Then, based on the acquired projection direction and angle of view or positional relationship, an image area to be allocated to each surface may be cut out, and audio information to be played back by the audio device 8 may be allocated. Although the audio device 8 is installed on the rear side of the surface 61a in Figs. 12(a) and (b), it may be installed, for example, on the rear side of another surface, or installed in the space 6. Furthermore, the audio device 8 may be configured to be a single audio device 8 by combining, for example, a plurality of audio modules (audio components, etc.).

[0106] According to the present invention, the first video acquisition unit 21 and the second video acquisition unit 23 acquire video including shooting information of live video and archive video, and the audio data acquisition unit 35 acquires audio information corresponding to the video. The distribution destination information may be appropriately acquired based on, in addition to the shooting information, for example, management information of each video or audio information, distribution request information, ranking information, past playback history information, audience information, etc., so that the most suitable distribution destination information is acquired. The distribution destination information may be acquired by being specified in advance in the shooting information, for example, or may be stored in the video storage unit 9.

[0107] According to the present invention, the first video acquisition unit 21 and the second video acquisition unit 23 acquire video including shooting information of live video and archive video, and the audio data acquisition unit 35 acquires audio information corresponding to the video. The distribution destination information may be appropriately acquired based on, in addition to the shooting information, for example, management information of each video or audio information, distribution request information, ranking information, past playback history information, audience information, etc., so that the most suitable distribution destination information is acquired. The distribution destination information may be acquired by being specified in advance in the shooting information, for example, or may be stored in the video storage unit 9.

[0108] Although the embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0109] 1. Image Region Generation System 2. Control device 3 Recording Module 5. Communication Network 6 Space 7 Display device (playback device) 8 Sound equipment (playback equipment) 9 Video storage unit 21 First moving image acquisition unit 23 Second moving image acquisition unit 25 Control section 26 Spatial information acquisition section 27 Judgment Department 28 Control Unit 31 Omnidirectional imaging device 32. Mike 35 Audio data acquisition unit 61 sides

Claims

1. An image area generating system for generating an image area to be displayed on each of rectangular faces surrounding a space, comprising: a video acquisition means for acquiring a video and distribution destination information for distributing the video; a determination means for determining a size ratio between the width and length of each of the surfaces based on an image of each of the surfaces captured by an imaging device installed in the space; an image area cutout means for cutting out a still image to be displayed into a plurality of image areas to be allocated to each of the planes based on the aspect ratio between the planes determined by the determination means and the distribution destination information acquired by the moving image acquisition means; an allocation means for allocating each image area cut out by the image area cutout means to each of the faces; An image region generating system comprising:

2. the moving image acquiring means acquires the distribution destination information including at least one of distributable contract information, distributable facility information, and distributable customer information regarding spectators; 2. The image region generating system according to claim 1, further comprising:

3. the moving image acquisition means acquires the distribution destination information including at least one of management information, distribution request information, ranking information, and past playback history information of each moving image or audio information; 2. The image region generating system according to claim 1, further comprising:

4. An image area generating system for generating an image area to be reproduced on each of rectangular faces surrounding a space, comprising: A video acquisition means for acquiring at least one of a live video and an archive video, audio information corresponding to the video, and distribution destination information for distributing the video and the audio information; a determination means for determining a size ratio between the width and length of each of the surfaces based on an image of each of the surfaces captured by an imaging device installed in the space; an image area cropping means for cropping each still image constituting the moving image acquired by the moving image acquisition means into a plurality of image areas to be allocated to each of the planes based on the layout relationship of the planes, the distribution destination information, and the aspect ratio between the planes determined by the determination means; an allocation means for determining a characteristic of each image area extracted by said image area extraction means and a characteristic of said space, and for allocating each of said image areas to each of said surfaces based on the determined characteristic of each image area and the characteristic of said space, and for allocating said acoustic information based on each of said allocated image areas; a data transmitting means for transmitting data including at least either the image areas or the acoustic information allocated by the allocating means to each reproduction device including at least either one of the display devices for reproducing the image areas on each of the surfaces or the acoustic devices for reproducing the acoustic information on each of the surfaces, through channels different from each other; An image region generation system comprising:

5. the image area extraction means extracts, based on the acoustic information, each still image constituting the moving image into a plurality of image areas in accordance with the arrangement relationship of each of the faces; 5. The image region generating system according to claim 4, further comprising:

6. said data transmitting means adjusts the image areas and the acoustic information of the data to be transmitted to achieve time-series synchronization with each other; 5. The image region generating system according to claim 4, further comprising:

7. An image area generating system for generating an image area to be reproduced on each of rectangular faces surrounding a space, comprising: A video acquisition means for acquiring at least one of a live video and an archive video, audio information corresponding to the video, and distribution destination information for distributing the video and the audio information; a determination means for determining a size ratio between the width and length of each of the surfaces based on an image of each of the surfaces captured by an imaging device installed in the space; an image area cropping means for cropping each still image constituting the moving image acquired by the moving image acquisition means into a plurality of image areas to be allocated to each of the planes based on the layout relationship of the planes, the distribution destination information, and the aspect ratio between the planes determined by the determination means; an extracting means for extracting spectator information consisting of one or more of the features of each image region extracted by the image region extracting means, and the positions, line of sight, head direction, and voices emitted by the spectators in the space; an allocation means for allocating each of the image regions to each of the faces and allocating the acoustic information based on each of the allocated image regions; a data transmitting means for transmitting data including at least either the image areas or the acoustic information allocated by the allocating means to each reproduction device including at least either one of the display devices for reproducing the image areas on each of the surfaces or the acoustic devices for reproducing the acoustic information on each of the surfaces, through channels different from each other; the moving image acquisition means resets imaging conditions for the live moving image based on the features of each image area extracted by the extraction means and on audience information; An image region generation system comprising:

8. In an image area display space in which an image area is reproduced on each of the rectangular faces surrounding the space, Each rectangular surface enclosing the space; a video acquisition means for acquiring a video and distribution destination information for distributing the video; a determination means for determining a size ratio between the width and length of each of the surfaces based on an image of each of the surfaces captured by an imaging device installed in the space; an image area cutout means for cutting out a still image to be displayed into a plurality of image areas to be allocated to each of the planes based on the aspect ratio between the planes determined by the determination means and the distribution destination information acquired by the moving image acquisition means; an allocation means for allocating each image area cut out by the image area cutout means to each of the faces; 13. An image area display device comprising:

9. An image area generating program for generating an image area to be displayed on each of rectangular faces surrounding a space, a moving image acquiring step of acquiring a moving image and distribution destination information to which the moving image is distributed; a determining step of determining a size ratio between the width and length of each of the surfaces based on an image of each surface captured by an imaging device installed in the space; an image area cutout step of cutting out a still image to be displayed into a plurality of image areas to be allocated to the respective screens based on the aspect ratio between the screens determined in the determination step and the distribution destination information acquired in the moving image acquisition step; an allocating step of allocating each image area extracted by the image area extracting step to each of the faces; 13. An image region generating program comprising:

Images