Information Processing Apparatus, Method for Generating Composite Video, and Computer-Readable Medium

Abstract

In a subject emphasizing device (20a) (information processing device), a visible light moving image acquisition unit (31) (first acquisition unit section) acquires a moving image including a subject (92) and a background, and a subject area extraction unit (33) extracts the area of the subject (92) from the moving image. Then, a background presentation unit (37) presents background information (94) in an area other than the area of the subject (92) in the moving image. An effect application unit (36) causes a video effect to act on the area of the subject (92) based on the intensity at which the video effect acts on the area of the subject (92), the intensity being determined by an effect intensity calculation unit (35) according to the distance from the camera to the subject (92) (information on the moving image acquisition environment). A video composition unit (38) composes the area of the subject (92) on which the effect application unit (36) has caused the video effect to act with the background information (94) presented by the background presentation unit (37) to generate a composite video (95).

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a method for generating a composite video, and a program. Background Art

[0002] In known conventional chroma key compositing, a portion of a video that has a specific color component and becomes transparent is combined with another video. Specifically, shooting is performed in a studio equipped with a green background screen, and an area having a color other than green is determined as a foreground (e.g., a subject) and extracted.

[0003] However, it is difficult to install a green background screen on the stage of a concert venue. In addition, since the light environment is greatly changed due to spotlights, fireworks, laser beams, etc. for guidance, it is difficult to extract the foreground at the live venue.

[0004] In response to such a problem, for example, Patent Document 1 proposes a foreground extraction technique using infrared rays.

[0005] Citation List

[0006] Patent Documents

[0007] Patent Document 1: JP 2000-23038A

[0008] Summary

[0009] Technical Problem

[0010] However, in Patent Document 1, it is described that the extracted foreground is combined with any background, but it is not described to combine a foreground to which an effect is being applied in real time with a background to enhance the visual effect.

[0011] The present disclosure provides an information processing apparatus, a method for generating a composite video, and a program configured to apply an effect to an extracted foreground having any background in real time.

[0012] Solution to Problem

[0013] To solve the above problems, an information processing apparatus according to an embodiment of the present disclosure includes: a first acquisition unit that acquires a moving image including a subject and a background; a subject area extraction unit that extracts an area of the subject from the moving image acquired by the first acquisition unit; a background drawing unit that draws background information in an area other than the area of the subject in the moving image acquired by the first acquisition unit; an effect application unit that applies an effect to the area of the subject based on an intensity of the effect applied to the area of the subject determined according to information about the environment in which the moving image is acquired; and a video synthesis unit that generates a composite video in which the area of the subject to which the effect is applied by the effect application unit is combined with the background information drawn by the background drawing unit. Description of the Drawings

[0014] Figure 1 is a block diagram schematically showing the configuration of a subject emphasis system according to the first embodiment.

[0015] Figure 2 is a hardware block diagram showing an example of the hardware configuration of the subject emphasis system.

[0016] Figure 3 is a functional block diagram showing an example of the functional configuration of a subject emphasis device according to the first embodiment.

[0017] Figure 4 is an exemplary external view of the mounting states of an RGB camera, an IR camera, and an IR lamp.

[0018] Figure 5 is a diagram showing the process of subject area extraction processing performed by the subject area extraction unit.

[0019] Figure 6 is a diagram showing an example of an image generated in the process of generating a composite video.

[0020] Figure 7 is a diagram showing an example of an effect applied by the effect application unit.

[0021] Figure 8 is a diagram showing the change in the magnitude of the difference between the position of a subject observed by the RGB camera and the position of the subject observed by the IR camera according to the shooting distance.

[0022] Figure 9 is a flowchart showing an example of the process of processing performed by the subject emphasis device according to the first embodiment.

[0023] Figure 10 is a functional block diagram showing an example of the functional configuration of a modified subject emphasis device according to the first embodiment.

[0024] Figure 11 is a diagram showing the schematic structure of an RGB-IR camera.

[0025] Figure 12 is a functional block diagram showing an example of the functional configuration of a subject emphasis device according to the second embodiment.

[0026] Figure 13 is a diagram showing the process of subject area extraction processing performed by the subject area extraction unit according to the second embodiment.

[0027] Figure 14It is a flowchart showing an example of the process executed by the subject emphasis device according to the second embodiment.

[0028] Figure 15 It is a functional block diagram showing an example of the functional configuration of the subject emphasis device according to the modification of the second embodiment.

[0029] Figure 16 It is a flowchart showing an example of the process executed by the subject emphasis device according to the modification of the second embodiment.

[0030] Figure 17 It is a functional block diagram showing an example of the functional configuration of the subject emphasis device according to the third embodiment.

[0031] Figure 18 It is a flowchart showing an example of the process executed by the subject emphasis device according to the third embodiment. Detailed Embodiment

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that in the following embodiments, the same parts are denoted by the same reference numerals and symbols, and their repeated description will be omitted.

[0033] In addition, the present disclosure will be described in the order of the items shown below.

[0034] 1. First Embodiment

[0035] 1-1. Configuration of Subject Emphasis System

[0036] 1-2. Hardware Configuration of Subject Emphasis System

[0037] 1-3. Functional Configuration of Subject Emphasis Device

[0038] 1-4. Description of Subject Area Extraction Process

[0039] 1-5. Description of Visual Effect

[0040] 1-6. Calculation of Effect Intensity

[0041] 1-7. Process of Processing Executed by Subject Emphasis Device

[0042] 1-8. Effects of First Embodiment

[0043] 1-9. Modification of First Embodiment

[0044] 1-10. Effects of Modification of First Embodiment

[0045] 2. Second Embodiment

[0046] 2-1. Functional Configuration of Subject Emphasis Device

[0047] 2-2. Description of Subject Region Extraction Processing

[0048] 2-3. Calculation of Effect Intensity

[0049] 2-4. Process of Processing Performed by Subject Emphasis Device

[0050] 2-5. Effects of Second Embodiment

[0051] 2-6. Modifications of Second Embodiment

[0052] 2-7. Effects of Modifications of Second Embodiment

[0053] 3. Third Embodiment

[0054] 3-1. Functional Configuration of Subject Emphasis Device

[0055] 3-2. Process of Processing Performed by Subject Emphasis Device

[0056] 3-3. Effects of Third Embodiment

[0057] (1. First Embodiment)

[0058] [1-1. Configuration of Subject Emphasis System]

[0059] First, an overview of the subject emphasis system 10a to which the present disclosure is applied will be described with reference to Figure 1 FIG. Figure 1 is a block diagram showing a schematic configuration of a subject emphasis system according to the first embodiment.

[0060] As Figure 1 shown, the subject emphasis system 10a includes an RGB camera 12, an IR camera 13, an IR lamp 14, a subject emphasis device 20a, and an LED panel 15.

[0061] The subject emphasis system 10a extracts a subject captured by the RGB camera 12 from the background. Then, a visual effect is applied to the region of the extracted subject, and then the region of the extracted subject is combined with the original background or another background. Note that in the present disclosure, the region of the subject may represent the entire subject or a part of the subject. For example, a region excluding a part of the main body of the subject (such as toes and hair) may be defined as the region of the subject. In addition, the visual effect is an example of the effects in the present disclosure.

[0062] The RGB camera 12 is a camera that captures the subject 92 on the stage 90. The RGB camera 12 is sensitive to visible light.

[0063] The IR camera 13 is an infrared camera that captures a range substantially equal to that of the RGB camera 12 and is sensitive to infrared rays, which are invisible light (e.g., a wavelength range of 800 nm to 900 nm). The IR camera 13 captures the subject 92 on the stage 90. Note that the relative positional relationship between the RGB camera 12 and the IR camera 13 is obtained by performing calibration in advance.

[0064] The IR lamp 14 is an infrared lamp that irradiates the shooting range of the IR camera 13. Note that the IR lamp 14 has an emission wavelength, and a large amount of light in the wavelength region to which the IR camera 13 is sensitive is included in its light. The IR lamp 14 includes, for example, an infrared light-emitting diode (LED) having an emission wavelength of approximately 800 nm to 900 nm.

[0065] The subject emphasizing device 20a extracts the region of each subject 92 from each of the video captured by the RGB camera 12 and the video captured by the IR camera 13. In addition, the subject emphasizing device 20a applies a visual effect to the extracted region of the subject 92. In addition, the subject emphasizing device 20a combines the region of the subject 92 to which the visual effect is applied with the region (background) other than the region of the subject 92 in the video captured by the RGB camera 12, and generates a composite video. Here, the background can directly use the region other than the subject 92 in the video captured by the RGB camera 12, or can use a region to which other information is applied. Note that the subject emphasizing device 20a is an example of the information processing device in the present disclosure.

[0066] The LED panel 15 is a display panel that is installed behind the stage 90, that is, behind the subject 92, and on which a plurality of LEDs are arranged vertically and horizontally. The LED panel 15 displays the composite video 95 output from the subject emphasizing device 20a. In the composite video 95, the subject 93 to which the visual effect is applied is combined with the background information 94 applied to the region other than the region of the subject 92 in the video captured by the RGB camera 12. This configuration enables the audience watching the stage 90 to watch both the performance of the subject 92 and the composite video 95 including the subject 93 to which the visual effect is applied and displayed on the LED panel 15. Note that the LED panel 15 has a surface with a low reflectivity, which suppresses the reflection of the irradiation light from the IR lamp 14 and the irradiation light irradiating the stage 90. This configuration enables the visibility of the composite video 95 displayed on the LED panel 15 to be enhanced. Therefore, this configuration enables the audience to simultaneously watch the subject 92 on the stage 90 and the composite video 95 displayed on the LED panel 15, thereby improving the stage effect.

[0067] Note that the subject emphasizing device 20a can send the composite video 95 to another place different from the stage 90. This configuration enables the composite video 95 to be watched in real time in another place.

[0068] [1-2. Hardware Configuration of the Subject Emphasis System]

[0069] Next, with reference to Figure 2 the hardware configuration of the subject emphasis system 10a will be described. Figure 2 is a hardware block diagram showing an example of the hardware configuration of the subject emphasis system.

[0070] The subject emphasis system 10a includes a central processing unit (CPU) 21, a read-only memory (ROM) 22, a random access memory (RAM) 23, a storage unit 24, and an input / output controller 25, which are connected to each other via an internal bus 26.

[0071] The control program stored in the storage unit 24 and various data files stored in the ROM 22 are loaded onto the RAM 23 and executed by the CPU 21, and thus the overall operation of the subject emphasis system 10a is controlled. In other words, the subject emphasis system 10a has the configuration of a general-purpose computer operated by a control program. Note that the control program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In addition, in the subject emphasis system 10a, a series of processing steps can be executed by hardware. Note that the control program executed by the CPU 21 can be a program that executes processing in time series according to the order described in the present disclosure, or can be a program that executes processing in parallel or at a necessary timing (e.g., when called).

[0072] The storage unit 24 includes, for example, a flash memory, and stores a control program and the like executed by the CPU 21.

[0073] The input / output controller 25 connects the CPU 21 and various input / output devices to input / output information.

[0074] The functions of the RGB camera 12, IR camera 13, IR lamp 14, and LED panel 15 as input / output devices connected to the input / output controller 25 are as described above. Note that for the input / output controller 25, an input device such as a keyboard through which an operator gives an operation instruction to the subject emphasis device 20a, a monitor device for monitoring the state of the subject emphasis device 20a, and a communication device for sending the composite video 95 output by the subject emphasis device 20a to the outside as needed.

[0075] [1-3. Functional Configuration of the Subject Emphasis Device]

[0076] Next, with reference to Figure 3 the functional configuration of the subject emphasis device 20a will be described. Figure 3This is a functional block diagram showing an example of the functional configuration of a subject emphasis device according to the first embodiment. The CPU 21 of the subject emphasis device 20a loads a control program onto the RAM 23 and runs the control program to implement Figure 3 each functional unit shown in

[0077] In other words, the CPU 21 of the subject emphasis device 20a implements a visible light moving image acquisition unit 31, an infrared moving image acquisition unit 32, a subject area extraction unit 33, a shooting distance calculation unit 34, an effect intensity calculation unit 35, an effect application unit 36, a background drawing unit 37, a video synthesis unit 38, and a display control unit 39 as functional units.

[0078] The visible light moving image acquisition unit 31 acquires a video signal captured by the RGB camera 12 to obtain an RGB video (moving image) including the subject 92 and the background. Note that the visible light moving image acquisition unit 31 is an example of the first acquisition unit in the present disclosure.

[0079] The infrared moving image acquisition unit 32 synchronizes with the timing of the visible light moving image acquisition unit 31 to acquire the video signal captured by the IR camera 13 at the timing of turning on the IR lamp 14, thereby obtaining an IR video (moving image) including the subject 92 and the background. Note that the infrared moving image acquisition unit 32 is an example of the second acquisition unit in the present disclosure.

[0080] The subject area extraction unit 33 extracts the area of the subject 92 from the RGB video acquired by the visible light moving image acquisition unit 31.

[0081] The shooting distance calculation unit 34 calculates the shooting distance when capturing the IR video acquired by the infrared moving image acquisition unit 32, that is, the distance from the IR camera 13 to the subject 92. Note that the shooting distance calculation unit 34 can calculate the distance from the RGB camera 12 to the subject 92. In the present disclosure, the shooting distance is an example of the environment for acquiring the moving image.

[0082] The effect intensity calculation unit 35 calculates the intensity of the visual effect applied to the area of the subject 92 extracted by the subject area extraction unit 33 from the RGB video acquired by the visible light moving image acquisition unit 31 based on the information about the environment for acquiring the RGB video acquired by the visible light moving image acquisition unit 31.

[0083] The effect application unit 36 applies the visual effect to the area of the subject 92 extracted by the subject area extraction unit 33 based on the intensity calculated by the effect intensity calculation unit 35.

[0084] The background drawing unit 37 draws background information 94 in the RGB video acquired by the visible light moving image acquisition unit 31 in areas other than the area of the subject 92. The background information 94 may be a live-action video or a computer graphics (CG) video. In addition, an image may be used instead of a video. Further, a plurality of pieces of background information 94 may be applied and drawn to a half area of the background obtained by, for example, vertical division, horizontal division, or inclined division.

[0085] The video composition unit 38 generates a composite video 95 by combining the area of the subject 92 to which the visual effect is applied by the effect application unit 36 and the background information 94 drawn by the background drawing unit 37.

[0086] The display control unit 39 causes the LED panel 15 to display the composite video 95 generated by the video composition unit 38. Note that the display control unit 39 is an example of the image display unit in the present disclosure.

[0087] [1-4. Description of the subject area extraction process]

[0088] Next, the content of the subject area extraction process performed by the subject area extraction unit 33 will be described with reference to Figures 4 to 6 the description. Figure 4 is an exemplary external view of the installation states of the RGB camera, the IR camera, and the IR lamp. Figure 5 is a diagram showing the process of the subject area extraction process performed by the subject area extraction unit. Figure 6 is a diagram showing an example of an image generated in the process of generating the composite video.

[0089] The RGB camera 12 and the IR camera 13 are installed at positions close to each other and face substantially the same direction. In other words, the RGB camera 12 and the IR camera 13 photograph substantially the same range. Then, the IR lamp 14 is installed near the IR camera 13. The IR lamp 14 includes, for example, a plurality of infrared LEDs that irradiate the photographing range of the IR camera 13. In Figure 4 the example, the IR lamp 14 is incorporated in the camera bracket 11 on which the RGB camera 12 and the IR camera 13 are installed. Note that a plurality of cameras having a Figure 4 configuration may be installed to surround the subject 92 to acquire a so-called volume video. Note that, when the cameras are installed to surround the subject 92, the cameras positioned in the diagonal direction desirably capture images at different times. This is because capturing images simultaneously by the cameras positioned in the diagonal direction causes noise due to the light from the IR lamp 14 positioned in the diagonal direction.

[0090] As Figure 5As shown, the RGB camera 12 and the IR camera 13 capture images synchronously with each other. The RGB video (moving image) captured by the RGB camera 12 is acquired by the visible light moving image acquisition unit 31 (image Ia in Figure 6 ). In addition, the IR video (moving image) captured by the IR camera 13 is acquired by the infrared moving image acquisition unit 32 (image Ib in Figure 6 ). Note that the processes described below are performed for each acquired image, and thus, the description will be made for the image rather than the moving image.

[0091] The subject area extraction unit 33 extracts the area of the subject 92 from the RGB image acquired by the visible light moving image acquisition unit 31 according to the process shown in Figure 5 . More specifically, the subject area extraction unit 33 extracts the area of the subject 92 from the IR image acquired by the infrared moving image acquisition unit 32, and extracts the area corresponding to the area of the subject 92 from the RGB image acquired by the visible light moving image acquisition unit 31.

[0092] First, the subject area extraction unit 33 calculates the difference (background difference 33a) between the IR image acquired by the infrared moving image acquisition unit 32 and the background image without the subject 92 captured by the IR camera 13 when the IR lamp 14 is turned on. The area of the subject 92 has a higher infrared reflectance than the infrared reflectance of the background image, and thus, when observed, it appears brighter in the IR image acquired by the infrared moving image acquisition unit 32. Therefore, the background difference 33a provides an IR image in which the area of the subject 92 is brightly emphasized.

[0093] Next, the subject area extraction unit 33 performs thresholding 33b on the IR image after calculating the background difference 33a. Thresholding 33b is a process of extracting only the pixels having pixel values indicating a brightness higher than a preset threshold. The area corresponding to the area of the subject 92 (mask) is extracted by thresholding 33b.

[0094] The area extracted in the thresholding 33b usually includes noise, and thus, the subject area extraction unit 33 performs noise removal 33c. For example, noise removal 33c is performed by removing isolated areas having an area equal to or less than a predetermined threshold.

[0095] In addition, the subject area extraction unit 33 performs a morphological operation 33d. The morphological operation 33d is a process performed on a binary image, such as a process of filling holes generated in the area. The morphological operation 33d provides the image Ic shown in Figure 6 .

[0096] Next, the subject area extraction unit 33 performs a warping 33e. The warping 33e is a process of moving the positions of the feature points specified in the moving image to transform them into another image (homography transformation). Specifically, the calibration information acquired in advance is used to predict how the position and shape of the area of the subject 92 extracted from the IR image captured by the IR camera 13 will appear in the RGB image captured by the RGB camera 12. The warping 33e provides the Figure 6 image Id shown in

[0097] At the same time, the subject area extraction unit 33 performs a demosaicing 33f on the RGB image captured by the RGB camera 12. The demosaicing 33f is a process of correcting the brightness of the image, for example.

[0098] Then, the background drawing unit 37 draws the background information 94 ( Figure 6 image Ie in

[0099] in the area outside the area (mask) of the subject 92 determined by the warping 33e. Figure 6 Then, the video synthesis unit 38 combines the background information 94 drawn by the background drawing unit 37 with the image captured by the RGB camera 12 that has undergone demosaicing by the subject area extraction unit 33. In this way, a synthesized video 95 (

[0100] image If in

[0101] [1-5. Description of Visual Effects]

[0102] Next, reference will be made to Figure 7 describe the visual effects applied by the effect application unit 36. Figure 7 is a diagram showing an example of the effects applied by the effect application unit.

[0103] The subject emphasis device 20a applies various visual effects to the area of the extracted subject 92. Thus, a video image of the subject 93 to which visual effects are applied is generated. Although the type of visual effect is not limited, an example thereof is shown in Figure 7

[0104] The visual effect E1 is used to blur the edges of the contour of the subject 92. The visual effect E1 provides an effect of making the errors in the extraction of the area of the subject 92 less obvious.

[0105] The visual effect E2 is used to output the silhouette of the subject 92. The visual effect E2 provides an effect of emphasizing the shape of the subject 92.​

[0106] The visual effect E3 is used to change the color or brightness of the edges of the contour of the subject 92. The visual effect E3 provides an effect of emphasizing the subject 92.

[0107] The visual effect E4 is used to display moving particles around the contour of the subject 92. The visual effect E4 provides an effect of emphasizing the contour of the subject 92.

[0108] The visual effect E5 is used to express the movement trajectory of the subject 92 by propagating the contour of the subject 92 extracted in the previous frame. The visual effect E5 provides an effect of emphasizing the movement of the subject 92.

[0109] Note that the types of visual effects are not limited to Figure 7 those shown in. For example, effects such as blurring the entire area of the subject 92 or changing the color of the area of the subject 92 can be applied.

[0110] [1-6. Calculation of effect intensity]

[0111] Next, with reference to Figure 8 the calculation of the intensity of the visual effect performed by the effect intensity calculation unit 35 will be described. Figure 8 is a graph showing how the magnitude of the difference in the position of the subject observed by the RGB camera and the position of the subject observed by the IR camera changes according to the shooting distance.

[0112] The RGB camera 12 and the IR camera 13 are placed at separate positions from each other. Therefore, a parallax ε appears in the positions of the subject 92 captured by the respective cameras. Therefore, the subject emphasizing device 20a performs calibration in advance and obtains the positional relationship between the RGB camera 12 and the IR camera 13.

[0113] Generally, calibration is performed by simultaneously acquiring images of the subject 92 by the RGB camera 12 and the IR camera 13, where the subject 92 is arranged at a predetermined position on the stage 90, and analyzing the displacement (parallax ε) of the position of the subject 92 acquired by the two cameras. In the calibration, the internal parameters and external parameters of the cameras are estimated. More specifically, for calibration, a calibration plate having a grid pattern thereon and held by the subject 92 is used, and calibration is performed by associating the position of the grid on the calibration plate observed by the RGB camera 12 with the position of the grid on the calibration plate observed by the IR camera 13.

[0114] In Figure 8 it is assumed that calibration is performed when the subject 92 is at the position P0 (the distance D between the camera and the subject 92 = D0). Additionally, it is assumed that the parallax ε is ε = ε0 at this time.

[0115] When the subject 92 moves from the position where calibration is performed, the parallax ε is different from the parallax at the time of performing calibration, and the RGB image acquired from the RGB camera 12 cannot be accurately aligned with the IR image acquired from the IR camera 13.

[0116] For example, assume that the subject 92 approaches the camera from the position where calibration is performed and moves to the position P1 (the distance D between the camera and the subject 92 = D1). At this time, the parallax ε = ε1 of the subject 92 between the images captured by the RGB camera 12 and the IR camera 13 is greater than the parallax ε = ε0 at the calibration position P0. In other words, ε1 > ε0 is satisfied. Therefore, even if the information obtained through calibration is used, the region of the subject 92 cannot be accurately extracted.

[0117] Meanwhile, assume that the subject 92 moves in the direction away from the camera from the position where calibration is performed and moves to the position P2 (the distance D between the camera and the subject 92 = D2). At this time, the parallax ε = ε2 of the subject 92 between the images captured by the RGB camera 12 and the IR camera 13 is less than the parallax ε = ε0 at the calibration position P0. In other words, ε2 < ε0 is satisfied. Therefore, even if the information obtained through calibration is used, the region of the subject 92 cannot be accurately extracted.

[0118] The effect intensity calculation unit 35 sets the intensity of the visual effect according to the distance (shooting distance) between the camera and the subject 92. Specifically, when the shooting distance is shorter than a predetermined distance, that is, when the subject 92 is closer than the calibration position (when the subject 92 is at a position where the parallax ε is greater than the parallax at the calibration position), a visual effect with an intensity higher than the intensity of the visual effect applied when the subject 92 is at the calibration position is applied. This configuration makes the positional displacement caused when extracting the region of the subject 92 less obvious. Note that a higher intensity of the visual effect means, for example, applying a stronger visual effect such as greatly blurring the edges around the subject 92.

[0119] On the other hand, when the shooting distance is longer than a predetermined distance, that is, when the subject 92 is at a position farther than the calibration position (when the subject 92 is at a position where the parallax ε is less than the parallax at the calibration position), it is preferable to apply a visual effect with an intensity equal to or less than the intensity of the visual effect applied when the subject 92 is at the calibration position.

[0120] Here, for example, the shooting distance calculation unit 34 calculates the distance (shooting distance) between the camera and the subject 92 based on the size (height) of the region of the subject 92 extracted from the infrared image captured by the IR camera 13. Note that a distance measurement unit such as a time-of-flight (TOF) sensor may be installed near the RGB camera 12 or the IR camera 13 so that the distance to the region of the subject 92 can be measured by the distance measurement unit.

[0121] [Process of the processing performed by the subject emphasis device]

[0122] The process of the processing performed by the subject emphasis device 20a will be described with reference to Figure 9 the following. Figure 9 FIG. is a flowchart showing an example of the process of the processing performed by the subject emphasis device according to the first embodiment. Note that Figure 9 the flowchart of shows the process of the processing performed after calibration for obtaining the positional relationship between the RGB camera 12 and the IR camera 13.

[0123] The visible light moving image acquisition unit 31 and the infrared moving image acquisition unit 32 simultaneously acquire an RGB image and an IR image with the IR lamp 14 turned on (step S11).

[0124] The subject area extraction unit 33 extracts the area of the subject 92 from the IR image acquired by the infrared moving image acquisition unit 32 (step S12).

[0125] The shooting distance calculation unit 34 calculates the shooting distance from the IR camera 13 to the subject 92 (step S13).

[0126] The effect intensity calculation unit 35 calculates the intensity of the visual effect based on the shooting distance calculated in step S13 (step S14).

[0127] The effect application unit 36 applies the visual effect to the area of the subject 92 to generate the area of the subject 93 to which the visual effect is applied (step S15).

[0128] The background drawing unit 37 generates and draws the background information 94 (step S16).

[0129] The video composition unit 38 generates a composite video 95 that combines the background information 94 and the area of the subject 93 to which the visual effect is applied (step S17).

[0130] The display control unit 39 causes the LED panel 15 to display the composite video 95 (step S18).

[0131] The visible light moving image acquisition unit 31 determines whether all images have been processed (step S19). When it is determined that all images have been processed (step S19: YES), the subject emphasis device 20a completes Figure 9 the processing. On the other hand, when it is not determined that all images have been processed (step S19: NO), the processing returns to step S11.

[0132] [1-8. Effects of the First Embodiment]

[0133] As described above, in the subject emphasis device 20a (information processing device) according to the first embodiment, the visible light moving image acquisition unit 31 (first acquisition unit) acquires an RGB video (moving image) including the subject 92 and the background, and the subject area extraction unit 33 extracts the area of the subject 92 from the moving image acquired by the visible light moving image acquisition unit 31. Then, the background drawing unit 37 draws the background information 94 in the area other than the area of the subject 92 in the moving image acquired by the visible light moving image acquisition unit 31. The effect application unit 36 applies a visual effect to the area of the subject 92 based on the intensity of the visual effect to be applied to the area of the subject 92, and the intensity of the visual effect is determined by the effect intensity calculation unit 35 according to the distance from the camera to the subject 92 (information about the environment in which the moving image is acquired). The video synthesis unit 38 combines the area of the subject 92 to which the visual effect has been applied by the effect application unit 36 with the background information 94 drawn by the background drawing unit 37 to generate a synthesized video 95.

[0134] This configuration enables the area of the subject 92 to be extracted from any background and the visual effect to be applied in real time. In addition, the boundary between the subject 92 and the background information 94 can be naturally shown.

[0135] Furthermore, in the subject emphasis device 20a (information processing device) according to the first embodiment, the effect application unit 36 applies an effect to the area of the subject 92 based on the intensity of the effect, and the intensity of the effect is determined by the effect intensity calculation unit 35 according to the shooting distance to the subject 92 calculated by the shooting distance calculation unit 34.

[0136] This configuration enables a visual effect for making errors less obvious to be applied when there is a possibility of errors occurring when extracting the area of the subject 92.

[0137] Furthermore, in the subject emphasis device 20a (information processing device) according to the first embodiment, when the shooting distance calculated by the shooting distance calculation unit 34 is shorter than the shooting distance (predetermined distance) at the time of calibration, the effect application unit 36 increases the intensity of the visual effect and applies the visual effect to the area of the subject 92.

[0138] This configuration enables the application of a visual effect that further makes the error less noticeable when an error may occur when extracting the region of the subject 92.

[0139] In addition, in the subject emphasizing device 20a (information processing device) according to the first embodiment, the infrared moving image acquisition unit 32 (second acquisition unit) acquires, in synchronization with the visible light moving image acquisition unit 31 (first acquisition unit), the intensity distribution (IR image) of the reflected light of the infrared light (invisible light) emitted to the subject 92 captured at a predetermined relative position of the RGB camera 12. Then, the subject region extraction unit 33 transforms the region of the subject 92 extracted from the IR image acquired by the infrared moving image acquisition unit 32 into the shape and position expected to be acquired by the visible light moving image acquisition unit 31, and applies the transformed region to the RGB image actually acquired by the visible light moving image acquisition unit 31, thereby extracting the region of the subject 92.

[0140] This configuration enables the accurate extraction of the region of the subject 92 through simple processing even when the surrounding light environment changes greatly.

[0141] In addition, in the subject emphasizing device 20a (information processing device) according to the first embodiment, the invisible light is infrared light.

[0142] This configuration enables the accurate extraction of the region of the subject 92 through simple processing without being affected by the surrounding light environment.

[0143] In addition, in the subject emphasizing device 20a (information processing device) according to the first embodiment, the display control unit 39 (video display unit) causes the composite video 95 from the video synthesis unit 38 to be displayed.

[0144] This configuration enables the combination of the region of the subject 93 to which the visual effect is applied and any background information 94.

[0145] In addition, in the subject emphasizing device 20a (information processing device) according to the first embodiment, the display control unit 39 (video display unit) displays the composite video 95 behind the subject 92.

[0146] This structure enables the viewer to view the subject 92 and the composite video 95 simultaneously.

[0147] In addition, in the subject emphasizing device 20a (information processing device) according to the first embodiment, the effect application unit 36 applies a visual effect to the periphery of the region of the subject 92.

[0148] This configuration enables the effective emphasis of the subject 92.

[0149] [1-9. Modification of the First Embodiment]

[0150] Next, a subject emphasis system 10b (not shown) according to a modification of the first embodiment will be described. Instead of the subject emphasis device 20a of the subject emphasis system 10a described in the first embodiment, the subject emphasis system 10b includes a subject emphasis device 20b.

[0151] Figure 10 It is a functional block diagram showing an example of the functional configuration of the subject emphasis device according to the modification of the first embodiment.

[0152] Instead of the shooting distance calculation unit 34 of the subject emphasis device 20a, the subject emphasis device 20b includes a parallax prediction unit 40. The parallax prediction unit 40 predicts the maximum value of the displacement between the actual position of the subject 92 and the result of the extraction of the subject 92 based on the movement range of the subject 92.

[0153] In the first embodiment, the subject emphasis device 20a has extracted the area of the subject 92 based on the positional relationship between the RGB camera 12 and the IR camera 13 obtained in advance by calibration. Calibration is performed using a pair of RGB images and IR images captured simultaneously in a state where the subject 92 is stationary at a predetermined position on the stage 90. Therefore, as described in the first embodiment, when the subject 92 moves on the stage 90, the area of the subject 92 cannot be accurately extracted.

[0154] However, the movement range of the subject 92 is limited to the range of the stage 90, and if the size of the stage 90 is known in advance, the range of the parallax ε between the subject 92 observed by the RGB camera 12 and the object 92 observed by the IR camera 13 can be estimated.

[0155] In the subject emphasis device 20b according to the modification of the first embodiment, the effect intensity calculation unit 35 calculates the intensity of the visual effect based on the maximum value of the parallax ε of the subject 92 predicted by the parallax prediction unit 40.

[0156] More specifically, regardless of the position of the subject 92, the effect intensity calculation unit 35 applies a visual effect having an intensity according to the maximum value of the parallax ε expected to occur when the subject 92 moves around on the stage 90.

[0157] In other words, when the subject 92 is located at the position where the parallax ε is maximum, a visual effect that makes the displacement of the position of the area of the subject 92 extracted by the subject emphasis device 20b not obvious is applied. Therefore, regardless of the position of the subject 92 on the stage 90, the displacement of the position of the area of the subject 92 can be made not obvious.

[0158] Although the description of the process of the processing performed by the subject emphasizing device 20b is omitted, in Figure 9 the flowchart of, instead of step S13, the parallax prediction unit 40 performs the process of calculating the maximum value of the parallax ε based on the information obtained by calibration and the size of the stage 90. Then, in step S14, the effect intensity calculation unit 35 calculates the intensity of the visual effect based on the maximum value of the parallax ε. Otherwise, the steps of the processing are the same as those of the processing shown in Figure 9 . In addition, these steps do not need to be performed for each image, and are preferably performed only once at the beginning.

[0159] [1-10. Effects of Modifications of the First Embodiment]

[0160] As described above, according to the subject emphasizing device 20b (information processing device) of the modification of the first embodiment, the effect application unit 36 applies the visual effect based on the maximum value of the parallax ε (displacement) between the actual position of the subject 92 and the result of the extraction of the subject 92 to the area of the subject 92, and the maximum value is predicted by the parallax prediction unit 40 based on the movement range of the subject 92.

[0161] This configuration enables the position displacement of the area of the subject 92 to be unobvious even when the subject 92 moves on the stage 90. In addition, the intensity of the visual effect is calculated only once when processing the first image, thereby providing high-speed processing.

[0162] (2. Second Embodiment)

[0163] The subject emphasizing system 10a described in the first embodiment includes two cameras (RGB camera 12 and IR camera 13), resulting in a positional displacement in the installation between the cameras, and thus calibration is required. The subject emphasizing system 10c (not shown) according to the second embodiment includes an RGB-IR camera 16 (see Figure 11 ), in which the RGB camera 12 and the IR camera 13 are integrated into one camera.

[0164] In another example of the environment for acquiring a moving image, the subject emphasizing system 10c (not shown) according to the second embodiment sets the intensity of the visual effect based on the difference in the processing time between the IR image and the RGB image, that is, the delay in the processing time.

[0165] [2-1. Functional Configuration of the Subject Emphasizing Device]

[0166] The subject emphasizing system 10c according to the second embodiment includes an RGB-IR camera 16, an IR lamp 14, a subject emphasizing device 20c, and an LED panel 15. In other words, in Figure 1In this case, the RGB camera 12 and the IR camera 13 are integrated into the RGB-IR camera 16, and the subject emphasis device 20a is replaced with the subject emphasis device 20c.

[0167] First, the structure of the RGB-IR camera 16 will be described with reference to Figure 11 FIG. Figure 11 FIG. is a diagram showing a schematic structure of the RGB-IR camera.

[0168] Figure 11 Two types of structures of the RGB-IR camera 16 are shown. The RGB-IR camera 16a is a camera including an imaging element 17 configured to simultaneously detect visible light and infrared light. The imaging element 17 has a structure in which an element sensitive to infrared light is incorporated in a part of the imaging element having a Bayer array. In other words, the imaging element 17 includes pixels IR sensitive to infrared light, and pixels R, G, and B for generating an RGB image. Therefore, the RGB-IR camera 16a is configured to acquire an RGB image and an IR image by one camera. There is no parallax between the acquired RGB image and IR image, and thus, the RGB image and the IR image can be aligned without using the relative positional relationship between the cameras described in the first embodiment.

[0169] In the RGB-IR camera 16b, the optical path of the incident light is divided into two by a beam splitter 18, and the visible light imaging element 19a and the infrared imaging element 19b are arranged in front of the divided optical paths. The beam splitter 18 has a structure in which two right-angled prisms are joined to each other.

[0170] Next, the functional configuration of the subject emphasis device 20c will be described with reference to Figure 12 FIG. Figure 12 FIG. is a functional block diagram showing an example of the functional configuration of the subject emphasis device according to the second embodiment. In other words, instead of the visible light moving image acquisition unit 31 and the infrared moving image acquisition unit 32 of the subject emphasis device 20a, the subject emphasis device 20c includes a moving image acquisition unit 41. In addition, instead of the subject area extraction unit 33, a subject area extraction unit 42 is provided. Further, instead of the shooting distance calculation unit 34, a processing time measurement unit 43 is provided.

[0171] The moving image acquisition unit 41 acquires the video signal captured by the RGB-IR camera 16 to obtain an RGB video (moving image) including the subject 92 and the background. In addition, the moving image acquisition unit 41 acquires the video signal captured by the RGB-IR camera 16 at the timing of turning on the IR lamp 14, thereby obtaining an IR video (moving image) including the subject 92 and the background. Note that the moving image acquisition unit 41 is an example of the first acquisition unit and the second acquisition unit in the present disclosure.

[0172] The subject area extraction unit 42 extracts the area of the subject 92 from the RGB video acquired by the moving image acquisition unit 41.

[0173] The processing time measurement unit 43 measures the elapsed time after the moving image is acquired by the moving image acquisition unit 41 (first acquisition unit).

[0174] [2-2. Description of subject area extraction processing]

[0175] Next, reference will be made to Figure 13 Describe the subject area extraction processing performed by the subject area extraction unit 42. Figure 13 is a diagram showing the process of subject area extraction processing performed by the subject area extraction unit according to the second embodiment.

[0176] The moving image acquisition unit 41 captures an RGB image and an IR image captured by the RGB-IR camera 16.

[0177] The subject area extraction unit 42 extracts the area of the subject 92 from the RGB image acquired by the moving image acquisition unit 41 according to Figure 13 the process shown in. More specifically, the subject area extraction unit 42 extracts the area of the subject 92 from the IR image acquired by the moving image acquisition unit 41, and extracts the area corresponding to the area of the subject 92 from the RGB image acquired by the moving image acquisition unit 41.

[0178] First, the subject area extraction unit 42 calculates the difference (background difference 42a) between the IR image acquired by the moving image acquisition unit 41 and the background image without the subject 92 captured by the RGB-IR camera 16 when the IR lamp 14 is turned on. The area of the subject 92 has a higher infrared reflectance than the background image, and thus appears brighter in the IR image acquired by the moving image acquisition unit 41 when observed. Therefore, the background difference 42a provides an IR image in which the area of the subject 92 is brightly emphasized.

[0179] Next, the subject area extraction unit 42 performs thresholding 42b on the IR image after calculating the background difference 42a. Thresholding 42b is a process of extracting only pixels having pixel values indicating luminance higher than a preset threshold. A region (mask) corresponding to the region of the subject image 92 is extracted by the thresholding 42b.

[0180] The region extracted by the thresholding 42b generally includes noise, and thus, the subject area extraction unit 42 performs noise removal 42c. For example, the noise removal 42c is performed by removing isolated regions having an area equal to or smaller than a predetermined threshold.

[0181] In addition, the subject area extraction unit 42 performs a morphological operation 42d. The morphological operation 42d is a process performed on a binary image, such as a process of filling holes generated in the region. Although the warping 33e was performed after the morphological operation in the first embodiment, there is no parallax between the RGB image and the IR image in this embodiment, and thus the warping is not necessary.

[0182] Meanwhile, the subject area extraction unit 42 performs demosaicing 42e on the RGB image captured by the RGB-IR camera 16. The demosaicing 42e is, for example, a process of correcting the luminance of the image.

[0183] Next, the background drawing unit 37 draws the background information 94 in a region other than the region (mask) of the subject 92 determined by the morphological operation 42d.

[0184] Then, the video synthesis unit 38 combines the background information 94 drawn by the background drawing unit 37 with the RGB image that has undergone demosaicing by the subject area extraction unit 42. In this way, a composite video 95 is generated in which the visible light image captured by the RGB-IR camera 16, that is, the image of the subject 92, is combined at a position corresponding to the region of the subject 92 in the background information 94.

[0185] [2-3. Calculation of effect intensity]

[0186] Next, the calculation of the intensity of the visual effect executed by the subject emphasis device 20c will be described. The process of the subject area extraction unit 42 extracting the area of the subject 92 from the IR image has a larger processing volume than the process of the RGB image. Therefore, the processing of the RGB image ends earlier than the processing of the IR image. When the delay of the time required for the processing of the IR image with respect to the time required for the processing of the RGB image is within a preset predetermined time, the processing can be continued in real time (for example, at the video rate). However, when the time required for the processing of the IR image increases, real-time processing cannot be continued, and a difference appears between the area of the subject 92 extracted from the IR image and the actual area of the subject 92 in the RGB image.

[0187] When the delay of the time required for the processing of the IR image with respect to the time required for the processing of the RGB image exceeds the predetermined time, the subject emphasis device 20c of the present embodiment sets the intensity of the visual effect stronger so that the position displacement of the area of the subject 92 is not obvious. More specifically, when the elapsed time measured by the processing time measurement unit 43 exceeds the predetermined time, the effect intensity calculation unit 35 calculates the intensity of the visual effect depending on whether the subject area extraction unit 42 has completed the extraction of the area of the subject 92.

[0188] In other words, the processing time measurement unit 43 measures the elapsed time after the moving image acquisition unit 41 acquires the RGB image and the IR image, and when the elapsed time exceeds the predetermined time, a visual effect with a higher intensity is applied. Note that the predetermined time is set to the time during which the subject area extraction unit 42 can perform the process of extracting the area of the subject 92 in real time (at the video rate).

[0189] [2-4. Process of the processing executed by the subject emphasis device]

[0190] will be referred to Figure 14 to describe the process of the processing executed by the subject emphasis device 20c. Figure 14 is a flowchart showing an example of the process of the processing executed by the subject emphasis device according to the second embodiment.

[0191] The moving image acquisition unit 41 acquires the RGB image and the IR image (step S21).

[0192] The subject area extraction unit 42 starts the extraction of the area of the subject 92 (step S22).

[0193] The processing time measurement unit 43 determines whether a predetermined time has elapsed since the image was acquired (step S23). When it is determined that the predetermined time has elapsed since the image was acquired (step S23: Yes), the processing proceeds to step S24. On the other hand, when it is not determined that the predetermined time has elapsed since the image was acquired (step S23: No), the processing proceeds to step S27.

[0194] If it is "Yes" in step S23, the subject area extraction unit 42 determines whether the extraction of the area of the subject 92 is completed (step S24). When it is determined that the extraction of the area of the subject 92 is completed (step S24: Yes), the processing proceeds to step S28. On the other hand, when it is not determined that the extraction of the area of the subject 92 is completed (step S24: No), the processing proceeds to step S25.

[0195] On the other hand, if it is "No" in step S23, the subject area extraction unit 42 determines whether the extraction of the area of the subject 92 is completed (step S27). When it is determined that the extraction of the area of the subject 92 is completed (step S27: Yes), the processing proceeds to step S28. On the other hand, when it is not determined that the extraction of the area of the subject 92 is completed (step S27: No), the processing returns to step S23.

[0196] Returning to step S24, if it is "No" in step S24, the subject area extraction unit 42 determines whether the extraction of the area of the subject 92 is completed (step S25). When it is determined that the extraction of the area of the subject 92 is completed (step S25: Yes), the processing proceeds to step S26. On the other hand, when it is not determined that the extraction of the area of the subject 92 is completed (step S25: No), step S25 is repeated.

[0197] If it is "Yes" in step S24 or step S27, the effect intensity calculation unit 35 sets the visual effect to a lower intensity (step S28). Then, the processing proceeds to step S29.

[0198] If it is "Yes" in step S25, the effect intensity calculation unit 35 sets the visual effect to a higher intensity (step S26). Then, the processing proceeds to step S29.

[0199] After step S26 or step S28, the effect application unit 36 applies a preset visual effect to the area of the subject 92 (step S29).

[0200] Next, the background drawing unit 37 generates and draws the background information 94 (step S30).

[0201] The video composition unit 38 generates a composite video 95 that combines the background information 94 with the area of the subject 93 to which the visual effect is applied (step S31).

[0202] The display control unit 39 causes the LED panel 15 to display the composite video 95 (step S32).

[0203] The moving image acquisition unit 41 determines whether all the images have been processed (step S33). When it is determined that all the images have been processed (step S33: YES), the subject emphasis device 20c finishes Figure 14 the processing. On the other hand, when it is not determined that all the images have been processed (step S33: NO), the processing returns to step S21.

[0204] [2-5. Effects of the Second Embodiment]

[0205] As described above, in the subject emphasis device 20c (information processing device) according to the second embodiment, when the elapsed time measured by the processing time measurement unit 43 from when the moving image acquisition unit 41 (first acquisition unit) acquires a moving image exceeds a predetermined time, the effect application unit 36 applies a visual effect to the area of the subject 92 based on the intensity of the visual effect determined depending on whether the subject area extraction unit 42 has completed the extraction of the area of the subject 92.

[0206] This configuration enables the position displacement of the area of the subject 92 in the composite video 95 to be unobvious even when it takes time to extract the area of the subject 92.

[0207] In addition, in the subject emphasis device 20c (information processing device) according to the second embodiment, if the subject area extraction unit 42 has not completed the extraction of the area of the subject 92 when the elapsed time measured by the processing time measurement unit 43 exceeds a predetermined time, the effect application unit 36 increases the intensity of the visual effect and applies the visual effect to the area of the subject 92.

[0208] This configuration enables an increased visual effect to be applied when it takes time to extract the area of the subject 92 and enables the position displacement of the area of the subject 92 in the composite video 95 to be unobvious.

[0209] [2-6. Modifications of the Second Embodiment]

[0210] Next, a subject emphasis system 10d (not shown) according to a modification of the second embodiment will be described. Instead of the subject emphasis device 20c described in the second embodiment, the subject emphasis system 10d includes a subject emphasis device 20d.

[0211] In the subject emphasis system 10d, a sudden change in the light environment caused by a spotlight on the stage 90, fireworks on the stage 90, etc. triggers the control of turning on / off the visual effect.

[0212] The turning on / off of the conventional visual effect is controlled based on a predetermined schedule, which requires troublesome prior settings. Meanwhile, the subject emphasis system 10d is configured to turn on / off the visual effect without presetting the turning on / off time of the visual effect.

[0213] Specifically, when the number of pixels with a luminance exceeding a predetermined luminance in the IR image acquired by the moving image acquisition unit 41 exceeds a predetermined value, the subject emphasis device 20d determines that a sudden change in the light environment (a large amount of infrared light has been emitted) has occurred, and turns on or off the visual effect.

[0214] Next, reference will be made to Figure 15 Describe the functional configuration of the subject emphasis device 20d. Figure 15 FIG. is a functional block diagram showing an example of the functional configuration of a subject emphasis device according to a modification of the second embodiment.

[0215] In addition to the functional components of the subject emphasis device 20c, the subject emphasis device 20d includes a trigger information detection unit 44 and an effect on / off control unit 45. The trigger information detection unit 44 counts the number of pixels with a luminance equal to or greater than a predetermined value in the IR image acquired by the moving image acquisition unit 41 (second acquisition unit).

[0216] The effect on / off control unit 45 switches whether to apply the visual effect according to the number of pixels counted by the trigger information detection unit 44, that is, whether to directly output the RGB image acquired by the moving image acquisition unit 41 or output the composite video 95 from the video composition unit 38. Note that the effect on / off control unit 45 is an example of the switching unit in the present disclosure.

[0217] Next, reference will be made to Figure 16 Describe the process of the process executed by the subject emphasis device 20d. Figure 16 FIG. is a flowchart showing an example of the process of the process executed by the subject emphasis device according to a modification of the second embodiment.

[0218] The moving image acquisition unit 41 acquires an RGB image and an IR image (step S41).

[0219] The subject area extraction unit 42 extracts the area of the subject 92 (step S42).

[0220] The trigger information detection unit 44 determines whether the number of pixels in the acquired IR image whose luminance is equal to or greater than a predetermined luminance is equal to or greater than a predetermined value (step S43). When it is determined that the number of pixels in the IR image whose luminance is equal to or greater than the predetermined luminance is equal to or greater than the predetermined value (step S43: Yes), the process proceeds to step S44. On the other hand, when it is not determined that the number of pixels in the IR image whose luminance is equal to or greater than the predetermined luminance is equal to or greater than the predetermined value (step S43: No), the process proceeds to step S45.

[0221] If it is "Yes" in step S43, the effect on / off control unit 45 stops applying the visual effect (step S44). Then, the process proceeds to step S47.

[0222] On the other hand, if it is "No" in step S43, the effect intensity calculation unit 35 sets the intensity of the visual effect according to the extraction delay of the region of the subject 92 (step S45).

[0223] Then, the effect application unit 36 applies the visual effect (step S46).

[0224] Next, the background drawing unit 37 generates and draws the background information 94 (step S47).

[0225] The video composition unit 38 generates a composite video 95 that combines the background information 94 with the region of the subject 93 to which the visual effect is applied (step S48).

[0226] The display control unit 39 causes the LED panel 15 to display the composite video 95 (step S49).

[0227] The moving image acquisition unit 41 determines whether all images have been processed (step S50). When it is determined that all images have been processed (step S50: Yes), the subject emphasizing device 20d completes Figure 16 the processing. On the other hand, when it is not determined that all images have been processed (step S50: No), the process returns to step S41.

[0228] Note that in the flowchart, an example in which a spotlight or fireworks trigger the turning off of the visual effect has been described. However, conversely, a spotlight or fireworks can trigger the turning on of the visual effect. In addition, a spotlight can be given only in the region where the pattern light is emitted by using a projector configured to project pattern light to turn on / off the visual effect.

[0229] Note that the spotlight timing can also be determined according to the excitement state of the on-site venue. In other words, for example, the sound pressure can be measured by a noise meter installed in the on-site venue, and the spotlight can be given based on the sound pressure reaching a predetermined sound pressure. This configuration enables the on / off control of the visual effect to be linked to the excitement of the on-site venue, thus further enhancing the stage effect.

[0230] In addition, a predetermined movement of the subject 92 can trigger the spotlight. The predetermined movement of the subject 92 can be detected by, for example, the method described later (third embodiment).

[0231] [2-7. Effects of Modifications of the Second Embodiment]

[0232] As described above, in the subject emphasis device 20d (information processing device) according to the modification of the second embodiment, the effect on / off control unit 45 (switching unit) switches whether to apply the visual effect based on the number of pixels in the IR image obtained by the moving image acquisition unit 41 (second acquisition unit) and having a brightness equal to or greater than a predetermined value, which is counted by the trigger information detection unit 44.

[0233] This configuration enables simple and reliable on / off control of the visual effect without the need to pre-set the time for performing the on / off control of the effect.

[0234] (3. Third Embodiment)

[0235] The subject emphasis system 10e (not shown) according to the third embodiment detects the movement of the detected and extracted subject 92, and sets the area to which the background information 94 is combined according to the detected movement.

[0236] Instead of the subject emphasis device 20a of the subject emphasis system 10a described in the first embodiment, the subject emphasis system 10e includes a subject emphasis device 20e.

[0237] [3-1. Functional Configuration of the Subject Emphasis Device]

[0238] First, reference will be made to Figure 17 Describe the functional configuration of the subject emphasis device 20e. Figure 17 is a functional block diagram showing an example of the functional configuration of the subject emphasis device according to the third embodiment.

[0239] The subject emphasis device 20e has a functional configuration in which a motion detection unit 46 and a subject synthesis position setting unit 47 are added to the subject emphasis device 20a described in the first embodiment (see Figure 3 ).

[0240] The motion detection unit 46 detects the motion of the subject 92. Specifically, the motion detection unit 46 detects the amplitude and direction of the motion by detecting, for example, the optical flow (motion vector) of the region of the subject 92 extracted by the subject region extraction unit 33 at different times.

[0241] Based on the motion of the subject 92 detected by the motion detection unit 46, the subject synthesis position setting unit 47 sets the position where the region of the subject 93 to which the visual effect is to be applied by the effect application unit 36 is combined with the background information 94 drawn by the background drawing unit 37. Specifically, the subject synthesis position setting unit 47 sets the position of the subject 92 (more precisely, the position of the subject 93 to which the visual effect is applied) to be combined with the background information 94 based on the amplitude and direction of the motion of the subject 92. Note that the subject synthesis position setting unit 47 is an example of the synthesis position setting unit in the present disclosure.

[0242] [3-2. Process of the processing performed by the subject emphasis device]

[0243] Next, the process of the processing performed by the subject emphasis device 20e will be described with reference to Figure 18 is a flowchart showing an example of the process of the processing performed by the subject emphasis device according to the third embodiment. Note that Figure 18 the flowchart of Figure 18 shows the process of the processing performed after calibration for obtaining the positional relationship between the RGB camera 12 and the IR camera 13.

[0244] The visible light motion image acquisition unit 31 and the infrared motion image acquisition unit 32 simultaneously acquire an RGB image and an IR image with the IR lamp 14 turned on (step S51).

[0245] The subject region extraction unit 33 extracts the region of the subject 92 from the IR image acquired by the infrared motion image acquisition unit 32 (step S52).

[0246] The motion detection unit 46 detects the motion of the subject 92 (step S53).

[0247] The subject synthesis position setting unit 47 calculates the region where the subject 92 is to be combined with the background information 94, that is, the background synthesis region (step S54).

[0248] Note that steps S55 to S61 are the same as the processes from steps S13 to S19 described in the first embodiment (see Figure 9 ), and thus the description thereof is omitted.

[0249] [3-3. Effects of the third embodiment]

[0250] As described above, in the subject emphasizing device 20e (information processing device) according to the third embodiment, the subject composition position setting unit 47 (composition position setting unit) sets the area where the subject 93 to which the visual effect is applied by the effect application unit 36 is combined with the background information 94 drawn by the background drawing unit 37, based on the movement of the subject 92 detected by the movement detection unit 46.

[0251] This configuration enables determination of the position of the subject 92 to be combined with the background information 94 by simple processing when the subject 92 has movement. In particular, the processing time required to detect the area of the subject 92 can be reduced for high-speed processing.

[0252] Although the present disclosure has been described according to several embodiments, these embodiments can be implemented by any device. The device preferably has necessary functional blocks to obtain necessary information.

[0253] In addition, for example, each step of a flowchart can be executed by one device, or can be executed by multiple devices. Further, when a single step includes multiple processes, the multiple processes can be executed by one device or can be executed by multiple devices. In other words, the multiple processes included in a single step can also be executed as a process including multiple steps. Conversely, a process described as multiple steps can be executed jointly as a single step.

[0254] In addition, for example, in a program executed by a computer, the processes describing the steps of the program can be executed in chronological order along the description in this specification, can be executed in parallel, or can be executed individually at a necessary timing such as a call. In other words, the processes of each step can be executed in an order different from the above order within a consistent range. Further, the processing steps describing the program can be executed in parallel with or in combination with the processing of another program.

[0255] In addition, for example, as long as there is no contradiction, each of the multiple technologies described in this specification can be independently implemented as a single technology. Of course, multiple arbitrary technologies can be combined with each other to be implemented. For example, part or all of the technology described in any embodiment can be implemented in combination with part or all of the technology described in another embodiment. In addition, part or all of any of the above technologies can be implemented in combination with another technology not described above.

[0256] It should be noted that the effects described herein are merely examples, and the present invention is not limited to these effects and can have other effects. The embodiments of the present disclosure are not limited to the above embodiments, and various modifications and changes can be made without departing from the spirit and scope of the present disclosure.

[0257] Note that the present disclosure may also have the following configurations. (1)

[0259] An information processing apparatus, comprising:

[0260] A first acquisition unit that acquires a moving image including a subject and a background;

[0261] A subject area extraction unit that extracts the area of the subject from the moving image acquired by the first acquisition unit;

[0262] A background drawing unit that draws background information in an area other than the area of the subject in the moving image acquired by the first acquisition unit;

[0263] An effect application unit that applies an effect to the area of the subject based on the intensity of the effect applied to the area of the subject determined according to information about the environment in which the moving image is acquired; and

[0264] A video synthesis unit that generates a synthesized video in which the area of the subject to which the effect has been applied by the effect application unit is combined with the background information drawn by the background drawing unit. (2)

[0266] The information processing apparatus according to (1), wherein

[0267] The effect application unit

[0268] Applies the effect to the area of the subject based on the intensity of the effect determined according to the shooting distance to the subject. (3)

[0270] The information processing apparatus according to (2), wherein

[0271] The effect application unit

[0272] When the shooting distance to the subject is less than a predetermined distance, increases the intensity of the effect and applies the effect to the area of the subject. (4)

[0274] The information processing apparatus according to any one of (1) to (3), wherein

[0275] The effect application unit

[0276] When the elapsed time from when the moving image is acquired by the first acquisition unit exceeds a predetermined time, the effect is applied to the area of the subject based on the intensity of the effect determined depending on whether the subject area extraction unit has completed the extraction of the area of the subject. (5)

[0278] The information processing apparatus according to (4), wherein,

[0279] the effect application unit

[0280] If the subject area extraction unit has not completed the extraction of the area of the subject when the elapsed time exceeds the predetermined time, the intensity of the effect is increased and the effect is applied to the area of the subject. (6)

[0282] The information processing apparatus according to any one of (1) to (5), further comprising:

[0283] a second acquisition unit that synchronously acquires, with the first acquisition unit, the intensity distribution of the reflected light of invisible light emitted to the subject and captured by the first acquisition unit at a predetermined relative position,

[0284] wherein the subject area extraction unit transforms the area of the subject extracted from the intensity distribution of the reflected light acquired by the second acquisition unit into the shape and position expected to be acquired by the first acquisition unit, applies the area to the moving image actually acquired by the first acquisition unit, and extracts the area of the subject. (7)

[0286] The information processing apparatus according to (6), wherein,

[0287] the invisible light is infrared light. (8)

[0289] The information processing apparatus according to any one of (1) to (7), further comprising:

[0290] a video display unit that displays the synthesized video from the video synthesis unit. (9)

[0292] The information processing apparatus according to (8), wherein,

[0293] the video display unit displays the synthesized video behind the subject. (10)

[0295] The information processing apparatus according to any one of (1) to (9), wherein,

[0296] The effect application unit applies the effect to the periphery of the area of the subject. (11)

[0298] The information processing apparatus according to any one of (1) to (10), wherein

[0299] the effect application unit

[0300] applies the effect, which is based on the maximum value of the displacement between the actual position of the subject and the extraction result of the subject, to the area of the subject, the maximum value being predicted based on the movement range of the subject. (12)

[0302] The information processing apparatus according to (6) or (7), further comprising:

[0303] a switching unit that switches whether to apply the effect according to the number of pixels having a luminance equal to or greater than a predetermined value in the intensity distribution of the reflected light acquired by the second acquisition unit. (13)

[0305] The information processing apparatus according to any one of (1) to (12), further comprising:

[0306] a composite position setting unit that sets an area where the area of the subject to which the effect is applied by the effect application unit and the background information drawn by the background drawing unit are to be combined, based on the movement of the subject. (14)

[0308] A composite video generation method, comprising:

[0309] a first acquisition step of acquiring a moving image including a subject and a background;

[0310] a subject extraction step of extracting an area of the subject from the moving image acquired in the first acquisition step;

[0311] a background drawing step of drawing background information in an area other than the area of the subject in the moving image acquired in the first acquisition step;

[0312] an effect application step of applying an effect to the area of the subject based on the intensity of the effect applied to the area of the subject determined according to information on the environment in which the moving image is acquired; and

[0313] A video synthesis step for generating a synthesized video, in which the region of the subject to which the effect is applied in the effect application step is combined with the background information drawn in the background drawing step. (15)

[0315] A program that causes a computer to function as:

[0316] A first acquisition unit that acquires a moving image including a subject and a background;

[0317] A subject region extraction unit that extracts the region of the subject from the moving image acquired by the first acquisition unit;

[0318] A background drawing unit that draws background information in the region other than the region of the subject in the moving image acquired by the first acquisition unit;

[0319] An effect application unit that applies an effect to the region of the subject based on the intensity of the effect applied to the region of the subject determined according to information about the environment in which the moving image is acquired; and

[0320] A video synthesis unit that generates a synthesized video, in which the region of the subject to which the effect is applied by the effect application unit is combined with the background information drawn by the background drawing unit.

[0321] List of reference numerals

[0322] 10a, 10b, 10c, 10d, 10e Subject emphasis system

[0323] 12 RGB camera (color camera)

[0324] 13 IR camera (infrared camera)

[0325] 14 IR lamp (infrared lamp)

[0326] 15 LED panel

[0327] 16 RGB-IR camera

[0328] 20a, 20b, 20c, 20d, 20e Subject emphasis device (information processing device)

[0329] 31 Visible light moving image acquisition unit (first acquisition unit)

[0330] 32 Infrared moving image acquisition unit (second acquisition unit)

[0331] 33, 42 Subject Area Extraction Unit

[0332] 34 Shooting Distance Calculation Unit

[0333] 35 Effect Intensity Calculation Unit

[0334] 36 Effect Application Unit

[0335] 37 Background Drawing Unit

[0336] 38 Video Composition Unit

[0337] 39 Display Control Unit (Video Display Unit)

[0338] 40 Parallax Prediction Unit

[0339] 41 Moving Image Acquisition Unit (First Acquisition Unit and Second Acquisition Unit)

[0340] 43 Processing Time Measurement Unit

[0341] 44 Trigger Information Detection Unit

[0342] 45 Effect On / Off Control Unit (Switching Unit)

[0343] 46 Motion Detection Unit

[0344] 47 Subject Composite Position Setting Unit (Composite Position Setting Unit)

[0345] 90 Stage

[0346] 92, 93 Subject

[0347] 94 Background Information

[0348] 95 Composite Video

[0349] ε Parallax

Claims

1. An information processing apparatus, comprising: A first acquisition unit that acquires a moving image including a subject and a background; A subject area extraction unit that extracts the area of the subject from the moving image acquired by the first acquisition unit; A background drawing unit that draws background information in an area other than the area of the subject in the moving image acquired by the first acquisition unit; An effect application unit that applies the effect to the area of the subject based on the intensity of the effect applied to the area of the subject determined according to information about the environment in which the moving image is acquired; And A video synthesis unit that generates a synthesized video in which the area of the subject to which the effect has been applied by the effect application unit is combined with the background information drawn by the background drawing unit; wherein the information processing device further includes a video display unit that displays the synthesized video from the video synthesis unit, and wherein the video display unit displays the synthesized video behind the subject.

2. The information processing apparatus according to claim 1, wherein, The effect application unit applies the effect to the area of the subject based on the intensity of the effect determined according to the shooting distance to the subject.

3. The information processing apparatus according to claim 2, wherein, The effect application unit increases the intensity of the effect and applies the effect to the area of the subject when the shooting distance to the subject is less than a predetermined distance.

4. The information processing apparatus according to claim 1, wherein, The effect application unit applies the effect to the area of the subject based on the intensity of the effect determined depending on whether the subject area extraction unit has completed the extraction of the area of the subject when the elapsed time since the moving image was acquired by the first acquisition unit exceeds a predetermined time.

5. The information processing apparatus according to claim 4, wherein, The effect application unit increases the intensity of the effect and applies the effect to the area of the subject if the subject area extraction unit has not completed the extraction of the area of the subject when the elapsed time exceeds the predetermined time.

6. The information processing apparatus according to claim 1, further comprising a second acquisition unit that acquires, synchronously with the first acquisition unit, an intensity distribution of reflected light of invisible light that is emitted to the subject and captured by the first acquisition unit at a predetermined relative position, wherein, The subject area extraction unit transforms the area of the subject extracted from the intensity distribution of the reflected light acquired by the second acquisition unit into the shape and position expected to be acquired by the first acquisition unit, applies the area to the moving image actually acquired by the first acquisition unit, and extracts the area of the subject.

7. The information processing apparatus according to claim 6, wherein, The invisible light is infrared light.

8. The information processing apparatus according to claim 1, wherein, The effect application unit applies the effect to the periphery of the area of the subject.

9. The information processing apparatus according to claim 1, wherein, The effect application unit applies the effect based on the maximum value of the displacement between the actual position of the subject and the extraction result of the subject to the area of the subject, and the maximum value is predicted based on the moving range of the subject.

10. The information processing apparatus according to claim 6, further comprising A switching unit that switches whether to apply the effect according to the number of pixels with a brightness equal to or greater than a predetermined value in the intensity distribution of the reflected light acquired by the second acquisition unit.

11. The information processing apparatus according to claim 1, further comprising A composite position setting unit that sets, based on the movement of the subject, an area where the area of the subject to which the effect is applied by the effect application unit is combined with the background information drawn by the background drawing unit.

12. A method for generating a composite video, comprising: A first acquisition step for acquiring a moving image including a subject and a background; A subject extraction step for extracting the area of the subject from the moving image acquired in the first acquisition step; A background drawing step for drawing background information in an area other than the area of the subject in the moving image obtained in the first acquisition step; An effect application step for applying the effect to the area of the subject based on the intensity of the effect applied to the area of the subject determined according to the information about the environment in which the moving image is acquired; And A video synthesis step for generating a synthesized video in which the area of the subject to which the effect is applied in the effect application step is combined with the background information drawn in the background drawing step, wherein the method for generating the synthesized video further includes displaying the synthesized video, and wherein the synthesized video is displayed behind the subject.

13. A computer-readable medium having a program recorded thereon, the program causing the computer to execute a method when executed by the computer, the method comprising: A first acquisition step for acquiring a moving image including a subject and a background; A subject extraction step for extracting the area of the subject from the moving image obtained in the first acquisition step; A background drawing step for drawing background information in an area other than the area of the subject in the moving image obtained in the first acquisition step; An effect application step for applying the effect to the area of the subject based on the intensity of the effect applied to the area of the subject determined according to the information about the environment in which the moving image is acquired; And A video synthesis step for generating a synthesized video in which the area of the subject to which the effect is applied in the effect application step is combined with the background information drawn in the background drawing step, wherein the method further includes displaying the synthesized video, and wherein the synthesized video is displayed behind the subject.

Images