Hall display system and event implementation method using the same

Abstract

To provide a hall display system that further enhances a realistic feeling and a method for executing an event using the hall display system.SOLUTION: A display system includes: a main screen 2 that displays an animation character operating in a hall H as a main character image MI; a sub-screen 3 that is arranged in the hall H separately from the main screen 2 and that can display video different from that on the main screen 2; a computer for generating video on the screen 2 and the sub-screen 3; and a first camera 11, which captures an audience seat AU of an audience watching both screens 2, 3. The character includes a second camera which is displayed as video on a part of the body. The range of the audience seat AU that falls within the virtual field of view of the second camera is selected as the selected video MSI from the video of the first camera 11, and the selected video MSI is displayed on the sub-screen 3.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a hall display system and a method for implementing an event using the same, and more particularly to a hall display system including a main screen that displays an animated character operating in the hall as a main character image, a sub-screen that is disposed in the hall separately from the main screen and is capable of displaying an image different from that of the main screen, and a computer that generates images for the main screen and the sub-screen, and a method for implementing an event using the same. [Background technology]

[0002] As an example of a hall display system that displays 3D images and a method for implementing an event using the same, there is one such as that disclosed in the following Patent Document 1. In this theater system, the aim is to achieve a sense of realism by combining a front screen and a rear screen of a hologram on the main screen. Meanwhile, in recent years, in pursuit of even greater realism, attempts have been made to improve the sense of realism by providing a sub-screen next to the main screen and, for example, displaying characters at different angles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-509027 Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the above-mentioned conventional situation, an object of the present invention is to provide a hall display system that further enhances the sense of realism, and a method for implementing an event using the same. [Means for solving the problem]

[0005] In order to achieve the above object, the hall display system according to the present invention is characterized in that it comprises a main screen which displays an animated character operating in the hall as a main character image, a sub-screen which is placed in the hall separately from the main screen and is capable of displaying an image different from that of the main screen, and a computer which generates images for the main screen and the sub-screen, the display system further comprises a first camera which photographs the audience seats of spectators watching both screens, the character has a second camera which is displayed as an image, the range of the character which falls within the virtual field of view of the second camera is generated as an image in the direction of the field of view of the second camera, and the generated image is displayed on the sub-screen, and further comprises a space in which an actor operates, a marker which is attached to the body of the actor, and a motion capture device which captures the movements of the actor with the marker, in a location not visible from the audience seats, the computer has a rendering function which generates in real time a character image which is linked to the captured movements of the actor, the image of the second camera is generated based on the marker of the actor which is located in the vicinity where the camera is displayed, and the virtual field of view is determined based on the marker.

[0006] According to the above characteristic configuration, the audience appears in the image of the auditorium captured by the first camera. Meanwhile, the character carries a second camera that is displayed as an image. Therefore, when the character makes a movement to capture the audience (audience) with the second camera that he or she carries, the range of the character that falls within the virtual field of view of the second camera is generated as an image in the field of view direction of the second camera. Then, the generated image is displayed on the sub-screen, which improves the sense of realism and increases the sense of unity between the character and the audience.

[0007] In the above configuration, the system further includes a space in which the actor moves, a marker attached to the actor's body, and a motion capture device that captures the actor's movements using the marker, the computer has a rendering function that generates in real time a character image that is linked to the captured actor's movements, the second camera generates an image based on the marker of the actor that exists in the vicinity of where the camera is displayed, and the virtual field of view is determined based on the marker. This reduces the sense of incongruity caused by the difference between the main character image and the selected image captured by the second camera of the character, increasing the reality of the selected image and further improving the sense of presence.

[0008] In any of the above configurations, the sub-screen may display an image of the audience in the virtual field of view of the second camera, and the range of the audience that falls within the virtual field of view of the second camera may be selected as a selected image from the image captured by the first camera, and the sub-screen may display both the image of the audience in the virtual field of view of the second camera and an image of the character that is included in the virtual field (so-called self-portrait image). In particular, the self-portrait image shows the audience together with the character, enhancing the sense of unity and liveliness.

[0009] In addition to any of the above configurations, it is preferable that the image of the audience displayed on the sub-screen is an image of a past time (T0-Δt) delayed by an infinitesimal time Δt from the real time T0 that is the reference for displaying the main character image, and the image on the sub-screen is an image of a past time (T0-Δt) delayed by an infinitesimal time Δt from the real time T0. This allows the audience to simultaneously see the main character image of the real time T0 and the selected image of the past time (T0-Δt). By simultaneously seeing the real time and the past time and the delayed image appearing on the sub-screen, people's attention shifts from the center to the sub-screen, which has the effect of making the audience feel a greater sense of spatial expansion and realism.

[0010] The main screen preferably comprises a hologram reproduction screen arranged at an angle above the stage for displaying hologram images, and a rear screen arranged behind the stage, and a composite image of both screens is displayed to the audience.

[0011] The sub-screens may be disposed at least to the left and right and / or above the main screen, thereby providing a left-right and / or right-left spread.

[0012] In addition, in order to achieve the above-mentioned object, a feature of the method of implementing an event according to the present invention is that the method of implementing an event using a hall display system comprising a main screen which displays an animated character operating in a hall as a main character image, a sub-screen which is arranged in the hall separately from the main screen and which is capable of displaying an image different from that of the main screen, and a computer which generates images for the main screen and the sub-screen, further comprises a first camera which photographs the audience seats of spectators watching both screens, the character has a second camera which is displayed as an image, the range of the character which falls within the virtual field of view of the second camera is generated as an image in the direction of the field of view of the second camera, and the generated image is displayed on the sub-screen, and further comprises, in a place not visible from the audience seats, a space in which an actor operates, a marker which is attached to the body of the actor, and a motion capture device which captures the movements of the actor with the marker, the computer has a rendering function which generates in real time a character image which is linked to the captured movements of the actor, the image of the second camera is generated based on the marker of the actor which is located in the vicinity where the camera is displayed, and the virtual field of view is determined based on the marker. Effect of the Invention

[0013] According to the features of the hall display system and the method for implementing an event using the same of the present invention described above, it is possible to provide a hall display system and a method for implementing an event using the same that provide an even greater sense of realism.

[0014] Other objects, configurations and effects of the present invention will become apparent from the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a perspective view showing the configuration of a hall according to the present invention. [Diagram 2] FIG. 2 is a side view showing a schematic configuration around the stage. [Diagram 3] FIG. 1 is a diagram showing a system configuration of the present invention. [Figure 4A] FIG. 13 is a diagram showing an image on a main screen. [Figure 4B] FIG. 13 is a diagram showing a captured image projected on a sub-screen. [Figure 4C] FIG. 13 is a diagram showing a self-portrait image displayed on a sub-screen. [Diagram 5] FIG. 13 is a diagram showing the relationship between the posture of a character on the main and sub screens and time. [Figure 6A] FIG. 13 shows the state of an actor when the character takes a photo of the audience. [Figure 6B] FIG. 13 is a diagram showing a character photographing the audience. [Figure 6C] A diagram showing the relationship between video of the audience and photographed images of characters. [Figure 7] FIG. 11 is a flow diagram for when a character takes a photo of the audience. [Figure 8A] FIG. 6B is a diagram equivalent to FIG. 6A when a character takes a selfie. [Figure 8B] FIG. 6B is a diagram equivalent to FIG. 6B when a character takes a selfie. [Figure 8C] FIG. 13 is a diagram showing a reflected image of a character. [Figure 9]FIG. 11 is a flow diagram for when a character takes a selfie. [Figure 10] FIG. 11 is a perspective view showing a second embodiment of the configuration in the hall. [Figure 11] FIG. 11 is a perspective view showing a third embodiment of the configuration in the hall. [Figure 12] FIG. 11 is a perspective view showing a fourth embodiment of the configuration in the hall. [Figure 13] FIG. 13(a) is a side view showing a schematic configuration around a stage according to a fifth embodiment, and (b) is a plan view of (a). [Figure 14] FIG. 13 is a plan view showing a schematic configuration around a stage according to a sixth embodiment. [Figure 15] FIG. 13 is a plan view showing the schematic configuration of the stage and audience seats according to a seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Next, the present invention will be described in more detail with reference to the accompanying drawings as appropriate. As shown in Figures 1 to 3, the hall display system 1 of the present invention roughly comprises a main screen 2, a pair of left and right sub-screens 3, lighting fixtures 4, speakers 6, a first camera 11 for capturing images of the audience seats AU of spectators watching both screens 2 and 3, and a computer 50 located in or near a hall H.

[0017] The hall display system 1 further includes a third camera 13 and a microphone 14 in a room separate from the hall H. The third camera 13 is installed in an isolated space S that cannot be seen from the audience seats AU, and captures the movements of an actor AC wearing multiple markers m on his body, and sends the images to a motion capture module MC51. The microphone 14 records the voice of a voice actor or vocalist and sends the images to a lip sync module LS54.

[0018] The main screen (main screen device) 2 installed in the hall H is equipped with a hologram reproduction screen 2a installed diagonally to the stage ST, a rear screen 2b installed behind the stage ST, a hologram projector 2c that projects an image onto the hologram reproduction screen 2a, and a rear projector 2d that projects an image onto the rear screen 2b. The image on the rear screen 2b passes through the hologram reproduction screen 2a and is composited with the image on the hologram reproduction screen 2a and is visible from the audience seats AU.

[0019] On the other hand, each sub-screen (sub-screen device) 3 is equipped with a first horizontal screen 3a1 and a second horizontal screen 3a2 (horizontal screen 3a) on the left and right, and a first sub-screen projector 3b1 and a second sub-screen projector (the latter is not shown in the figure, and both are collectively referred to as the sub-screen projector 3b) that project images onto these horizontal screens 3a, respectively.

[0020] The first camera 11 is, for example, disposed (fixed) facing (directly facing) the audience seats AU at a position closer to the audience seats AU than the hologram reproduction screen 2a and the rear screen 2b, and captures a range (field of view) that includes the entire audience seats AU. Since the entire audience seats AU is captured from directly in front of the audience seats AU, differences and distortions from the audience seats AU image BSI, which will be described later, are unlikely to occur in the selected image MSI, and the sense of incongruity of the selected image MSI is also eliminated. In addition, since the first camera 11 covers most of the audience seats AU, the portion (range) of the audience seats AU that is not captured in the selected image is reduced, so that many of the audience members can be given the opportunity to be captured in the selected image, and the sense of realism and unity is also increased. For example, a wide-angle camera is used as the first camera 11. In addition, in the present invention, the character CH has a virtual camera CA as a second camera and performs an operation of capturing the audience seats AU.

[0021] The computer 50 includes a motion capture module MC51, a real-time rendering module RR52, a video synthesis module IM53, a lip sync module LS54, a backstage image generation module BI55, a virtual light image generation module VL56, a real light controller RC58, a music generation module SG59, and a time management controller TM60 that serves as the basis for synchronizing the time of each module.

[0022] The motion capture module MC51 analyzes the movements of the actor AC captured by the third camera 13, and generates a 3D image of the moving character in the real-time rendering module RR52. Multiple third cameras 13 are provided, and capture the actor AC from different angles to perform motion capture. Usually, the shooting direction of the front third camera 13 becomes the basic viewing direction D1 when generating the main character.

[0023] The virtual light image generation module VL56 creates an image of lighting in the virtual space, and in conjunction with the real light controller RC58, sends light information to the backstage image generation module BI55 and the real-time rendering module RR52 to generate a shading-equipped image consistent with the lighting conditions. The real lighting control unit RL61 is controlled by the real light controller RC58 and operates the real lighting fixtures 4 to perform real lighting in the hall H.

[0024] The image BSI of the audience seats AU captured by the first camera 11 is trimmed (selected) by the image synthesis module IM53 within the range of the virtual visual field VS of the virtual camera CA, and projected by the sub-screen projector 3b on the horizontal screen 3a as a captured image MSI1 captured by the character CH. In the case of a selfie, which will be described later, the image of the audience seats AU trimmed in the same manner as above (sub-background image BSI2, which will be described later) is further synthesized with a sub-character image SI of the real-time rendering module RR52, and projected by the sub-screen projector 3b on the horizontal screen 3a as a selfie image MSI2 of the character CH. Meanwhile, the image of the real-time rendering module RR52 is projected as a main character image MI on the hologram playback screen 2a and the rear screen 2b by the hologram projector 2c and rear projector 2d, respectively.

[0025] The lip sync module LS54 delays the voice of the voice actor or the like recorded by the microphone 14 to match the image of the main character. In addition, the music generation module SG59 generates background music and sound effects, and these background music and the like are sent to the hall H by the speaker 6 via the sound system 62 together with the voice of the microphone 14.

[0026] As shown in Fig. 1 and 5, characters CH (CH-A with hair, CH-B without hair) are displayed on the main screen 2. In contrast, as shown in Fig. 1, 4B and C, a pair of left and right sub-screens 3 display images MSI1 and MSI2 of the characters CH (CH-A, CH-B) captured by a virtual camera CA as a selected image MSI. Note that, although the example shown in the figure shows different images being displayed on the pair of left and right sub-screens 3, it is of course possible to display the same selected image MSI.

[0027] To explain the image on the main screen 2 in detail, as shown in Fig. 4A, the main character image MI is composed of the main character body image MI1 and the main character's shadow MI2. Furthermore, the main background image BMI is added to this to create the main composite image MMI.

[0028] On the other hand, in the example shown in Fig. 4B, an image of the audience seats AU taken by the character CH is displayed as a shot image MSI1 on the first horizontal screen 3a1. Also, in the example shown in Fig. 4C, a self-shot image MSI2 taken by the character CH with the audience seats AU in the background is displayed on the second horizontal screen 3a2. In this self-shot image MSI2, the sub-character image SI is made up of a sub-character main body image SI1 and a sub-character shadow SI2. Then, by further adding a sub-background image (part of the image of the audience seats AU) BSI2, the self-shot image MSI2 is created.

[0029] The main composite image MMI on the main screen 2 uses a character rendered in a basic direction view D1. This basic direction view D1 may or may not match the direction view of the third camera 13. On the other hand, the direction view D2 different from the basic direction view D1 may be, for example, the direction view D3 of the first camera 11. Therefore, the character CH taking the selfie faces backwards to the audience seats AU on the main screen 2, but faces forward on the sub-screen 3.

[0030] Now, the generation of the photographed image MSI1 of the character CH will be described in detail with reference to FIGS. When the character CH (CH-A) points the second camera, which is a virtual camera CA, in front of his / her body toward the audience AU, only the audience AU is captured in the image. The image of the first camera 11 is used for the image BSI of the audience AU (steps S11 and S12).

[0031] For example, when a character CH (CH-A) takes a picture by holding a smartphone SP in his / her hand as a second camera, which is a virtual camera CA, as shown in Figs. 6A and 6B, the position (x, y, z) of the virtual camera CA is calculated from the set dimensions of the smartphone SP and the character CH based on the position (x0, y0, z0) of the marker m0 on the arm of the actor AC, which is closest to the virtual camera CA. The actor AC does not need to actually hold the smartphone. Then, by providing a virtual field of view VS to the position of the camera CA, a trimming range TR (TR1) in the image BSI of the first camera 11 is determined as shown in Fig. 6C. The trimming range TR is planar position information of the image BSI, and is therefore expressed by the x coordinate (width (horizontal) direction of the stage ST) and the z coordinate (vertical direction) (step S13). Then, the image MSI1 (step S14) trimmed based on this planar position information is displayed on the sub-screen 3 (step S15).

[0032] Next, the generation of the selfie image MSI2 of the character CH (CH-B) will be described in detail with reference to FIGS. When the character CH takes a selfie with his / her back facing the audience AU, the audience AU and part of the character CH's body are captured in the image. As for the sub background image BSI2 as the image of the audience AU, an image obtained by trimming (steps S11 to S14) the image BSI from the first camera 11 in the trimming range TR (TR2) in Fig. 6C in the same manner as described above is used.

[0033] Here, the position (x', y', z') of the camera CA is calculated based on the position (x0', y0', z0') of the reference marker m0' as described above (step S21). In addition, the position information of the reflected body is three-dimensional position information including the position information of the arm, face, and body in the example of FIG. 8C, and is obtained by adding the y coordinate (front-back direction of the stage ST) to the previous x coordinate and z coordinate (steps S22 and S23). This position information is based on the position information of each marker m of the actor AC. Then, based on these three-dimensional position information, a portion (range) of the character CH reflected in the virtual field of view VS of the virtual camera CA is determined (step S24), and the portion is rendered to generate a sub-character image SI (step S25). Then, the self-portrait image MSI2 (step S26) in which the trimmed sub-background image BSI2 and the rendered sub-character image SI of the character CH are combined is displayed on the sub-screen 3 (step S27).

[0034] In FIG. 1, real illumination light L1 is emitted from a real lighting device 4 onto an audience seating AU, a stage ST, and a character CH.

[0035] As shown in Figures 1, 3 to 5, the character CH at real time T0 is displayed on the main screen 2. In contrast, images MSI1 and 2 captured by the character CH at a time (T0-Δt) going back an infinitesimal time Δt are displayed on the sub-screen 3. Shadows and backgrounds are generated for the images on each screen based on information about real illumination light L1 from a real lighting fixture 4 that is set for each time.

[0036] The images BSI1,2 of the audience seats AU in the images MSI1,2 also use images from a past time (T0-Δt) going back an infinitesimal time Δt. In particular, in the case of the selfie image MSI2, the sub-character image SI also uses an image from a past time (T0-Δt), so there are no inconsistencies such as different lighting colors, and the audience accepts the image on the sub-screen 3 with a very natural feeling, resulting in a natural sense of realism. In the example of Figure 5, the character CH at real time (T0) is in the middle of turning his body from the selfie pose at the past time (T0-Δt) to the audience seats AU, and the character CH surrounded by the dashed circle in the figure is projected on the screen.

[0037] Incidentally, the images BSI1,2 of the audience seating AU may be delayed by a minute time Δt due to the performance of the hardware. In the case of the selfie image MSI2, the sub-character image SI may use a past image going back by the minute time Δt in accordance with the performance of the hardware, and the minute time Δt may be adjusted by adjusting the hardware delay and the software delay of the image synthesis module IM53. The minute time Δt is preferably set to 0.1 to 0.5 seconds for optimal implementation. If it is 0.1 seconds or more, the difference in the images can be recognized, and if it exceeds 0.5 seconds, it will appear unnatural.

[0038] A main character main body image MI1, a main character shadow MI2, and a main background image BMI are synthesized and displayed on the main screen 2. These elements can be displayed on the hologram reproduction screen 2a and the rear screen 2b in the following combinations. Case 1) Hologram playback screen 2a; main character body image MI1, rear screen 2b; main character shadow MI2 and main background image BMI Case 2) Hologram playback screen 2a: Main character body image MI1 and main character shadow MI2, rear screen 2b: Main background image BMI Case 3) Hologram playback screen 2a; main character body image MI1, main character shadow MI2 and part of main background image BMI (BMI1), rear screen 2b; remainder of main background image BMI (BMI2) The screen layout of these images will be adapted as appropriate according to need.

[0039] In the above example, a space for the actor AC to move and a motion capture device are provided. However, as shown in Figure 3, the event may be carried out by recording all of the controls for the video, audio, lighting, etc. in the event in the recording module RS57 and playing them back.

[0040] Furthermore, selected video such as the selfie video MSI2 recorded in the recording module RS57 may be sent (distributed) to, for example, a mobile device of a spectator in the audience seating AU, or posted (transmitted) in real time on a network such as a social networking service (SNS). This allows a sense of unity to be created with fans other than the spectators in the audience seating AU.

[0041] Next, another embodiment of the present invention will be described. The same members as those in the first embodiment are designated by the same reference numerals. In the above embodiment, the virtual camera CA is a smartphone SP, but the present invention is not limited to this and can be applied to various types of photographing devices. Also, although an example in which the smartphone SP is held in the hand is shown, it may be another part of the character CH, or a photographing method using another photographing tool such as a selfie stick may be used.

[0042] In the above embodiment, the first camera 11 is placed between the audience seats AU and the stage ST. However, the position of the first camera 11 is not limited to this as long as it can capture almost the entire audience seats AU. For example, the first camera 11 may be fixed above or behind the main screen 2 (hologram reproduction screen 2a) so as to capture the entire audience seats AU from directly in front of the audience seats AU. In this position, the position of the first camera 11 and the position of the character CH projected on the main screen 2 are almost the same relative to the audience seats AU, increasing the reality of the self-shot image MSI2. However, it is necessary to ensure that the character CH projected on the main screen 2 is not reflected in the first camera 11.

[0043] In the above embodiment, a pair of left and right horizontal screens 3a (3a1, 3a2) are provided as the sub-screens 3, but as shown in FIG. 10, an upper screen 3c may be additionally provided above the main screen 2. In this example, the image in the trimming range TR3 of FIG. 6c is used. Also, as shown in FIG. 11, instead of providing a pair of left and right horizontal screens 3a (3a1, 3a2), a pair of left and right upper screens 3c (3c1, 3c2) may be provided. Each of these upper screens 3c is provided with a corresponding projector, but this is omitted from the illustration and will be the same hereinafter.

[0044] In the above embodiment, the main composite image MMI projected on the main screen 2 is the real time T0, and the selected images MSI projected on the left and right sub-screens are the past time (T0-Δt), and the time difference between the screens is set to two stages. However, the time between the three screens may be changed to three stages. For example, the real time may be used for the main screen 2, the future time for one sub-screen 3, and the past time for the other sub-screen 3, or multiple times such as the present, past, and further past may be used. The number of screens and the switching of the time may be three or more.

[0045] In another embodiment of Fig. 12, in addition to the pair of left and right horizontal screens 3a (3a1, 3a2), a pair of left and right upper screens 3c (3c1, 3c2) are provided. The pair of left and right upper screens 3c (3c1, 3c2) display past images (MSI1', MSI2') from time T0-2Δt, and the pair of left and right horizontal screens 3a (3a1, 3a2) display past images (MSI1, MSI2) from time T0-Δt. In addition, the times of the above horizontal screens 3a and upper screens 3c can be reversed, all sub-screens can be set to the same time, or the time of the sub-screens can be changed and selected as appropriate.

[0046] In another embodiment of Fig. 13, a further pair of the above-mentioned main screen 2 and sub-screen 3 are provided back-to-back on the stage ST. For example, the hologram reproduction screen 2a is provided with a first set of members whose final alphabet is a and a second set of members whose final alphabet is b, as indicated by the symbols 2aa and 2ab. In addition, a pair of left and right first horizontal screens 3aa (3a1a, 3a2a) and a pair of left and right second horizontal screens 3ab (3a1b, 3a2b) are provided with screen drive devices 70 that move the horizontal screens up, down, left and right, respectively, and move the sub-screen 3a up, down and left and right in accordance with the sound and image.

[0047] In another embodiment of FIG. 14, four sets of the above-mentioned main screen 2 and sub-screen 3 (only 3c) are provided on a stage ST, back-to-back and shifted by 90 degrees. In the first to fourth sets, the last alphabet of the code is a to d to identify each set. In this arrangement, the audience can see the screens 2 and 3 from a 360-degree range around, and by providing the stage ST in the center of an arena, for example, it is possible to seat more audience members closer to the main screen 2. In this case, in addition to projecting images at the same time on the first to fourth sets, the first to fourth sets may project images at different times.

[0048] 15, in addition to the pair of left and right first horizontal screens 3a (3a1, 3a2) provided beside the stage ST, a pair of left and right second horizontal screens 3ax (3a1x, 3a2x) and a third horizontal screen 3ay (3a1y, 3a2y) may be provided in sequence closer to the audience seats 80. The time of the images on each of these horizontal screens can be changed as appropriate according to the combinations described above. [Industrial Applicability]

[0049] The present invention can be used as a hall display system that can be used for concerts and events featuring virtual characters, and as a method for implementing events using the system. The hall may be outdoors, and any hall may be used as long as a screen can be installed on the stage. [Explanation of symbols]

[0050] AU: Audience, CH (CH-A, CH-B, CH-C): Character, H: Hall, ST: Stage, AC: Actor, S: Isolated space, D1: View in basic direction, D2: View in direction different from basic direction, D3: View in direction of first camera, MI (MIa, MIb): Main character image, MI1: Main character body image, MI2: Shadow of main character, BMI: Main background image, MMI: Main composite image, SI: Sub character image, SI1: Sub character main character image, SI2: shadow of sub-character, BSI: image of audience, MSI (MSI1, MSI2, MSI1', MSI2'): selected image, MSI1: recorded image, MSI2: self-shot image, L1: real lighting, T0: real time, Δt: minute time, T0-Δt: past time, m0: reference marker, m: marker, VS: virtual field of view, CA: virtual camera (second camera), TR: trimming range, 1: hall display system, 2: main screen, 2a: hologram playback screen 1: 1st horizontal screen, 2b: rear screen, 2c: hologram projector, 2d: rear projector, 3: sub-screen, 3a: horizontal screen, 3a1: first horizontal screen, 3a2: second horizontal screen, 3b: sub-screen projector, 3b1: first sub-screen projector, 4: real lighting fixture, 6: speaker, 11: first camera, 13: third camera, 14: microphone, 50: computer, MC51: motion capture module, RR52: real-time rendering module, IM53: video synthesis module, LS54: lip sync module, BI55: backstage image generation module, VL56: virtual light image generation module, VS (VS1, VS2, VS3): virtual field of view, RS57: recording module, RC58: real light controller, SG59: music generation module, TM60: time management controller, RL61: real lighting control unit, 62: sound system

Claims

1. A main screen that displays the animated characters moving in the hall as main character images; a sub-screen that is disposed in the hall separately from the main screen and is capable of displaying an image different from that of the main screen; A hall display system including a computer that generates images of the main screen and the sub-screen, A first camera is further provided for photographing the audience seating area of ​​the audience watching both screens, The character has a second camera that is displayed as an image, A range of the character that falls within a virtual field of view of the second camera is generated as an image in a field of view direction of the second camera; Displaying the generated image on the sub-screen; The system further includes a space for an actor to move in a place not visible from the audience seats, a marker to be attached to the body of the actor, and a motion capture device to capture the movement of the actor by the marker, The computer has a rendering function that generates character images in real time that correspond to the captured actor's movements, A hole display system in which the second camera is generated as an image based on a marker of the actor located in the vicinity of where the camera is displayed, and the virtual field of view is determined based on the marker.

2. 2. A display system for a hall according to claim 1, wherein the sub-screen displays an image of the audience seats in a virtual field of view of the second camera.

3. A display system for a hall as described in claim 1, wherein the range of the audience seats that falls within the virtual field of view of the second camera is selected as a selected image from the image captured by the first camera, and the sub-screen displays both the image of the audience seats in the virtual field of view of the second camera and the image of the character included in the virtual field of view.

4. A hall display system as described in any one of claims 1 to 3, wherein the image of the audience seats displayed on the sub-screen is an image from a past time (T0-Δt) delayed by a small time Δt from the real time T0 that is the reference for displaying the main character image, and the image on the sub-screen is an image from a past time (T0-Δt) delayed by a small time Δt from the real time T0.

5. The main screen comprises a hologram playback screen arranged at an angle above the stage for displaying hologram images, and a rear screen arranged behind the stage, and displays a composite image of both screens to the audience.

6. 6. A hall display system according to claim 1, wherein the sub-screen is disposed at least to the left and right and / or above the main screen.

7. A main screen that displays the animated characters moving in the hall as main character images; a sub-screen that is disposed in the hall separately from the main screen and is capable of displaying an image different from that of the main screen; A method for implementing an event using a hall display system including the main screen and a computer for generating images on the sub-screen, comprising: A first camera is further provided for photographing the audience seating area of ​​the audience watching both screens, The character has a second camera that is displayed as an image, A range of the character that falls within a virtual field of view of the second camera is generated as an image in a field of view direction of the second camera; The generated image is displayed on the sub-screen, and the system further includes a space for an actor to move, a marker attached to the body of the actor, and a motion capture device that captures the movement of the actor using the marker, in a location not visible from the audience seats; The computer has a rendering function that generates character images in real time that are linked to the captured actor's movements, A method for implementing an event in which the second camera is generated as an image based on a marker of the actor located in the vicinity in which the camera is displayed, and the virtual field of view is determined based on the marker.

Images