Information processing device, synthetic image generating method and program

Abstract

A visible light moving image acquisition unit (31) (first acquisition unit) in a subject emphasis device (20a) (information processing device) acquires a moving image including a subject (92) and a background, and a subject region extraction unit (33) extracts a region of the subject (92) from the moving image. Then, a background rendering unit (37) renders background information (94) in a region other than the region of the subject (92) in the moving image. An effect application unit (36) causes a video effect to act on the region of the subject (92) on the basis of strength of the video effect to act on the region of the subject (92), the strength being determined by an effect strength calculation unit (35) in accordance with a distance from a camera to the subject (92) (information on moving image acquisition environment). A video combining unit (38) combines the region of the subject (92) on which the effect application unit (36) has caused the video effect to act and the background information (94) rendered by the background rendering unit (37), to generate combined video (95).

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a synthetic video generating method, and a program. [Background technology]

[0002] Conventionally, chromakey compositing is known, in which a part of an image of a specific color component is made transparent and another image is composited into the transparent part. Specifically, shooting is done in a studio with a green screen, and the area other than green is judged to be the foreground (for example, the subject) and extracted.

[0003] However, it is difficult to install a green screen on the stage at a live concert venue, and the lighting environment changes dramatically due to spotlights, fireworks, laser lights, and other effects used for the performance, making it difficult to extract the foreground.

[0004] To address this issue, for example, Patent Document 1 proposes a technique for extracting foreground using infrared rays. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2000-23038 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, although Patent Document 1 mentions that the extracted foreground can be composited with any background, it does not mention that the visual effect can be enhanced by applying effects to the extracted foreground in real time and compositing it with the background.

[0007] The present disclosure proposes an information processing device, a synthetic video generating method, and a program that can apply effects in real time to a foreground extracted from an arbitrary background. [Means for solving the problem]

[0008] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure includes a first acquisition unit that acquires a moving image including a subject and a background, a subject region extraction unit that extracts a region of the subject from the moving image acquired by the first acquisition unit, a background drawing unit that draws background information in a region other than the region of the subject in the moving image acquired by the first acquisition unit, an effect application unit that applies an effect to the region of the subject based on an intensity of the effect to be applied to the region of the subject determined according to information related to an acquisition environment of the moving image, and an image synthesis unit that generates a synthetic image by synthesizing the region of the subject to which the effect application unit has applied an effect and the background information drawn by the background drawing unit. the effect application unit applies the effect to the area of ​​the subject based on the strength of the effect determined depending on whether or not the subject area extraction unit has completed extraction of the area of ​​the subject when a predetermined time has elapsed since the first acquisition unit acquired the video. It is an information processing device. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a schematic configuration of a subject emphasizing system according to a first embodiment. [Diagram 2] FIG. 1 is a hardware block diagram showing an example of a hardware configuration of a subject emphasizing system. [Diagram 3] 1 is a functional block diagram showing an example of a functional configuration of an object emphasizing device according to a first embodiment. [Figure 4] FIG. 2 is an external view showing an example of an installation state of an RGB camera, an IR camera, and an IR light. [Diagram 5] 11 is a diagram showing a flow of a subject region extraction process performed by a subject region extraction unit. FIG. [Figure 6] FIG. 13 is a diagram showing an example of an image generated in the process of generating a composite video. [Figure 7] 5A to 5C are diagrams showing examples of effects applied by an effect application section. [Figure 8]11 is a diagram illustrating how the magnitude of the deviation between the position of the subject observed by an RGB camera and the position of the subject observed by an IR camera changes depending on the shooting distance. FIG. [Figure 9] 5 is a flowchart showing an example of the flow of processing performed by the subject emphasizing device of the first embodiment. [Figure 10] FIG. 2 is a functional block diagram showing an example of a functional configuration of a subject emphasizing device according to a modified example of the first embodiment. [Figure 11] FIG. 1 is a diagram illustrating a schematic structure of an RGB-IR camera. [Figure 12] FIG. 11 is a functional block diagram showing an example of a functional configuration of an object emphasizing device according to a second embodiment. [Figure 13] FIG. 11 is a diagram showing a flow of a subject region extraction process performed by a subject region extraction unit of the second embodiment. [Figure 14] 10 is a flowchart showing an example of the flow of processing performed by an object emphasizing device according to a second embodiment. [Figure 15] FIG. 13 is a functional block diagram showing an example of a functional configuration of a subject emphasizing device according to a modified example of the second embodiment. [Figure 16] 13 is a flowchart showing an example of the flow of processing performed by an object emphasizing device according to a modified example of the second embodiment. [Figure 17] FIG. 13 is a functional block diagram showing an example of the functional configuration of an object emphasizing device according to a third embodiment. [Figure 18] 13 is a flowchart showing an example of the flow of processing performed by an object emphasizing device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] The present disclosure will be described in the following order. 1. First embodiment 1-1. Configuration of subject emphasis system 1-2.Hardware configuration of the subject enhancement system 1-3. Functional configuration of the subject enhancement device 1-4. Description of subject area extraction process 1-5.Explanation of video effects 1-6. Calculating effect strength 1-7. Processing flow of the subject enhancement device 1-8. Effects of the First Embodiment 1-9. Modification of the first embodiment 1-10. Effects of the Modification of the First Embodiment 2. Second embodiment 2-1. Functional configuration of the subject enhancement device 2-2. Description of subject area extraction process 2-3. Calculating effect strength 2-4. Processing flow of the subject enhancement device 2-5. Effects of the Second Embodiment 2-6. Modification of the second embodiment 2-7. Effects of the Modification of the Second Embodiment 3. Third embodiment 3-1. Functional configuration of the subject enhancement device 3-2. Processing flow of the subject enhancement device 3-3. Advantages of the Third Embodiment

[0012] 1. First Embodiment [1-1. Configuration of subject enhancement system] First, an overview of an object emphasizing system 10a to which the present disclosure is applied will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of an object emphasizing system according to a first embodiment.

[0013] As shown in FIG. 1, the subject highlighting system 10a includes an RGB camera 12, an IR camera 13, an IR light 14, an subject highlighting device 20a, and an LED panel 15.

[0014] The subject highlighting system 10a extracts the subject captured by the RGB camera 12 from the background. Then, a video effect is applied to the extracted area of ​​the subject, and the extracted area is composited with the original background or another background. In this disclosure, the area of ​​the subject may mean the entire subject, or a part of the subject. For example, the area of ​​the subject excluding parts of the subject's body (such as the toes and hair) may be the area of ​​the subject. In addition, the video effect is an example of an effect in this disclosure.

[0015] The RGB camera 12 is a color camera that captures an image of a subject 92 on a stage 90. The RGB camera 12 has sensitivity in the visible light range.

[0016] The IR camera 13 is an infrared camera having sensitivity to the infrared region (for example, a region of wavelengths of 800 to 900 nm) which is an invisible light region, and captures an area approximately equal to that of the RGB camera 12. The IR camera 13 captures an image of a subject 92 on a stage 90. The relative positional relationship between the RGB camera 12 and the IR camera 13 is acquired in advance by performing calibration.

[0017] The IR light 14 is an infrared light that illuminates the shooting range of the IR camera 13. The emission wavelength of the IR light 14 includes a large amount of light in the wavelength range to which the IR camera 13 is sensitive. The IR light 14 is, for example, an infrared LED (Light Emitting Diode) with an emission wavelength of about 800 to 900 nm.

[0018] The subject emphasizing device 20a extracts the area of ​​the subject 92 from the image captured by the RGB camera 12 and the image captured by the IR camera 13. The subject emphasizing device 20a also applies a video effect to the extracted area of ​​the subject 92. The subject emphasizing device 20a further generates a composite image by combining the area of ​​the subject 92 to which the video effect has been applied with an area (background) other than the area of ​​the subject 92 in the image captured by the RGB camera 12. Here, the background may be the area other than the subject 92 in the image captured by the RGB camera 12 itself, or may be an area in which other information is fitted. The subject emphasizing device 20a is an example of an information processing device in the present disclosure.

[0019] The LED panel 15 is a display panel with multiple LEDs arranged vertically and horizontally, installed at the back of the stage 90, i.e., behind the subject 92. The LED panel 15 displays a composite image 95, which is a composite of the subject 93 to which a video effect has been applied, output by the subject emphasizing device 20a, and background information 94 that has been fitted to an area other than the subject 92 in the image captured by the RGB camera 12. This allows the audience watching the stage 90 to view the performance of the subject 92 and the composite image 95 displayed on the LED panel 15, including the subject 93 to which a video effect has been applied. Note that the reflectance of the surface of the LED panel 15 is kept low, so that reflection of the illumination light from the IR light 14 and the illumination light illuminating the stage 90 is suppressed. This makes it possible to improve the visibility of the composite image 95 displayed on the LED panel 15. The audience can simultaneously view the subject 92 on the stage 90 and the composite image 95 displayed on the LED panel 15, thereby enhancing the production effect.

[0020] The subject emphasizing device 20a may transmit the composite video 95 to a location separate from the stage 90. In this case, the composite video 95 can be viewed in real time at the separate location.

[0021] [1-2.Hardware configuration of the subject enhancement system] Next, the hardware configuration of the subject highlighting system 10a will be described with reference to Fig. 2. Fig. 2 is a hardware block diagram showing an example of the hardware configuration of the subject highlighting system.

[0022] The subject highlighting system 10a has a configuration in which a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, and an input / output controller 25 are connected via an internal bus .

[0023] The CPU 21 controls the overall operation of the subject enhancement system 10a by expanding the control program stored in the storage unit 24 and various data files stored in the ROM 22 on the RAM 23 and executing them. That is, the subject enhancement system 10a has a configuration of a general computer operated by a control program. The control program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. The subject enhancement system 10a may also execute a series of processes by hardware. The control program executed by the CPU 21 may be a program in which processing is performed in chronological order according to the sequence described in this disclosure, or may be a program in which processing is performed in parallel or at a required timing such as when called.

[0024] The storage unit 24 is configured by, for example, a flash memory, and stores the control program executed by the CPU 21 and the like.

[0025] The input / output controller 25 connects the CPU 21 to various input / output devices, and inputs and outputs information.

[0026] The functions of the RGB camera 12, IR camera 13, IR light 14, and LED panel 15, which are input / output devices connected to the input / output controller 25, are as described above. Note that, if necessary, the input / output controller 25 may be connected to an input device such as a keyboard through which an operator issues operation instructions to the subject emphasizing device 20a, a monitor device that monitors the state of the subject emphasizing device 20a, and a communication device that transmits the composite image 95 output by the subject emphasizing device 20a to the outside.

[0027] [1-3. Functional configuration of the subject enhancement device] Next, the functional configuration of the object emphasizing device 20a will be described with reference to Fig. 3. Fig. 3 is a functional block diagram showing an example of the functional configuration of the object emphasizing device of the first embodiment. The CPU 21 of the object emphasizing device 20a realizes each functional unit shown in Fig. 3 by expanding a control program on the RAM 23 and running it.

[0028] That is, the CPU 21 of the subject highlighting device 20a realizes, as functional units, a visible light video acquisition unit 31, an infrared video acquisition unit 32, a subject area extraction unit 33, a shooting distance calculation unit 34, an effect intensity calculation unit 35, an effect application unit 36, a background drawing unit 37, an image synthesis unit 38, and a display control unit 39.

[0029] Visible light video capture section 31 captures an RGB video (video) including subject 92 and the background by acquiring a video signal captured by RGB camera 12. Visible light video capture section 31 is an example of a first capture section in this disclosure.

[0030] The infrared video acquisition unit 32 acquires an IR image (video) including the subject 92 and the background by acquiring a video signal captured by the IR camera 13 at a timing when the IR light 14 is turned on, in synchronization with the timing at which the visible light video acquisition unit 31 acquires a video. The infrared video acquisition unit 32 is an example of a second acquisition unit in this disclosure.

[0031] The subject region extraction section 33 extracts the region of the subject 92 from the RGB image acquired by the visible light moving image acquisition section 31 .

[0032] Shooting distance calculation unit 34 calculates the shooting distance of the IR video captured by infrared video capture unit 32, i.e., the distance from IR camera 13 to subject 92. Note that shooting distance calculation unit 34 may also calculate the distance from RGB camera 12 to subject 92. The shooting distance is an example of a video capture environment in this disclosure.

[0033] The effect intensity calculation unit 35 calculates the intensity of the video effect to be applied to the area of ​​the subject 92 extracted by the subject area extraction unit 33 in the RGB video acquired by the visible light video acquisition unit 31, based on information relating to the acquisition environment of the RGB video acquired by the visible light video acquisition unit 31.

[0034] The effect application section 36 applies a video effect to the area of ​​the subject 92 extracted by the subject area extraction section 33 based on the intensity calculated by the effect intensity calculation section 35 .

[0035] Background drawing section 37 draws background information 94 in an area other than the area of ​​subject 92 in the RGB image acquired by visible light video acquisition section 31. Background information 94 may be a real-life image or a CG (Computer Graphics) image. It may also be an image instead of a video. A plurality of pieces of background information 94 may be divided into areas, such as half vertically, half horizontally, or half diagonally, and drawn.

[0036] The video composition section 38 generates a composite video 95 by combining the area of ​​the subject 92 to which the effect application section 36 has applied the video effect and the background information 94 drawn by the background drawing section 37 .

[0037] The display control unit 39 causes the composite image 95 generated by the image composition unit 38 to be displayed on the LED panel 15. Note that the display control unit 39 is an example of the image display unit in this disclosure.

[0038] [1-4. Description of subject area extraction process] Next, the contents of the subject region extraction process performed by the subject region extraction unit 33 will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is an external view showing an example of the installation state of the RGB camera, the IR camera, and the IR light. Fig. 5 is a diagram showing the flow of the subject region extraction process performed by the subject region extraction unit. Fig. 6 is a diagram showing an example of an image generated in the process of generating a composite video.

[0039] The RGB camera 12 and the IR camera 13 are installed in close proximity to each other and facing approximately the same direction. That is, the RGB camera 12 and the IR camera 13 capture approximately the same range. An IR light 14 is installed near the IR camera 13. The IR light 14 is composed of, for example, a plurality of infrared LEDs, and illuminates the capture range of the IR camera 13. In the example of FIG. 4, the IR light 14 is incorporated in the camera stand 11 on which the RGB camera 12 and the IR camera 13 are installed. Note that a plurality of cameras having the configuration of FIG. 4 may be installed to surround the subject 92, and a so-called volumetric image may be obtained. Note that, when the cameras are installed to surround the subject 92, it is preferable that the cameras in the diagonal direction capture images at different times. This is because, if the cameras in the diagonal direction capture images simultaneously, the light of the IR light 14 in the diagonal direction becomes noise.

[0040] As shown in Fig. 5, the RGB camera 12 and the IR camera 13 capture images in a synchronized state. The RGB video (video) captured by the RGB camera 12 is acquired by the visible light video acquisition unit 31 (image Ia in Fig. 6). The IR video (video) captured by the IR camera 13 is acquired by the infrared video acquisition unit 32 (image Ib in Fig. 6). Note that the processing described below is performed on a per-acquired image basis, and therefore will be described using the term "image" instead of "video."

[0041] 5, subject region extraction section 33 extracts the region of subject 92 from the RGB image acquired by visible light video acquisition section 31. More specifically, subject region extraction section 33 extracts the region of subject 92 from the IR image acquired by infrared video acquisition section 32, and extracts a region in the RGB image acquired by visible light video acquisition section 31 that corresponds to the region of subject 92.

[0042] First, subject region extraction section 33 calculates the difference (background difference 33a) between the IR image acquired by infrared video acquisition section 32 and a background image captured by IR camera 13 with IR light 14 turned on and without subject 92 present. The region of subject 92 has a higher infrared reflectance than the background image, and is therefore observed as bright in the IR image acquired by infrared video acquisition section 32. Therefore, by background difference 33a, an image is obtained in which the region of subject 92 is brightly emphasized in the IR image.

[0043] Next, the subject region extraction unit 33 performs threshold processing 33b on the IR image that has been subjected to background subtraction 33a. The threshold processing 33b is a process for extracting only pixels having pixel values ​​brighter than a preset threshold. The threshold processing 33b extracts a region (mask) corresponding to the region of the subject 92.

[0044] Since the region extracted by the threshold process 33b generally contains noise, the subject region extraction section 33 performs noise removal 33c. The noise removal 33c is performed, for example, by removing isolated regions whose area is equal to or smaller than a predetermined threshold.

[0045] Furthermore, the subject region extraction unit 33 performs a morphology process 33d. The morphology process 33d is a process performed on a binary image, for example, to fill holes that occur in the region. By the morphology process 33d, an image Ic shown in FIG. 6 is obtained.

[0046] Next, the subject region extraction unit 33 performs a warping process 33e. The warping process 33e is a process of moving the positions of feature points specified in the image and transforming the image into another image (homography transformation). Specifically, using calibration information acquired in advance, it predicts how the position and shape of the region of the subject 92 extracted from the IR image captured by the IR camera 13 will appear in the RGB image captured by the RGB camera 12. An image Id shown in FIG. 6 is obtained by the warping process 33e.

[0047] Meanwhile, the subject region extraction unit 33 performs a demosaic process 33f on the RGB image captured by the RGB camera 12. The demosaic process 33f is, for example, a process of correcting the brightness of the image.

[0048] Next, background drawing section 37 draws background information 94 in the area other than the area (mask) of subject 92 determined by warping process 33e (image Ie in FIG. 6).

[0049] Then, the image synthesis unit 38 synthesizes the background information 94 drawn by the background drawing unit 37 and the image of the RGB camera 12 on which the subject region extraction unit 33 has performed demosaic processing. As a result, a synthetic image 95 is generated in which the visible light image captured by the RGB camera 12, i.e., the image of the subject 92, is synthesized at a position of the background information 94 corresponding to the region of the subject 92 (image If in FIG. 6).

[0050] Then, the display control unit 39 causes the LED panel 15 to display the composite image 95.

[0051] [1-5. Explanation of video effects] Next, the video effects applied by the effect application section 36 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of an effect applied by the effect application section.

[0052] The subject emphasizing device 20a applies various video effects to the extracted region of the subject 92. This generates a video of the subject 93 to which the video effects have been applied. Any type of video effect may be used, and an example is shown in FIG.

[0053] The video effect E1 blurs the edges of the outer periphery of the subject 92. The video effect E1 has the effect of making extraction errors in the area of ​​the subject 92 less noticeable.

[0054] The video effect E2 outputs the silhouette of the subject 92. The video effect E2 has the effect of making the shape of the subject 92 stand out.

[0055] The video effect E3 changes the color or brightness of the outer edge of the subject 92. The video effect E3 has the effect of making the subject 92 stand out.

[0056] The video effect E4 displays moving particles on the outer periphery of the subject 92. The video effect E4 has the effect of making the contours of the subject 92 stand out.

[0057] Video effect E5 propagates the contour of subject 92 extracted in the previous frame to express the movement trajectory of subject 92. Video effect E5 has the effect of emphasizing the movement of subject 92.

[0058] It should be noted that the types of video effects are not limited to those shown in Fig. 7. For example, effects such as blurring the entire area of ​​subject 92 or changing the color of the area of ​​subject 92 may be applied.

[0059] [1-6. Calculation of effect strength] Next, the calculation of the intensity of a video effect performed by the effect intensity calculation unit 35 will be described with reference to Fig. 8. Fig. 8 is a diagram illustrating how the magnitude of the deviation between the position of a subject observed by an RGB camera and the position of a subject observed by an IR camera changes depending on the shooting distance.

[0060] Since the RGB camera 12 and the IR camera 13 are placed at positions apart from each other, a parallax ε occurs between the positions of the subject 92 photographed by each camera. Therefore, the subject emphasizing device 20a acquires the positional relationship between the RGB camera 12 and the IR camera 13 by performing calibration in advance.

[0061] In general, calibration is performed by simultaneously acquiring images of the subject 92 with the RGB camera 12 and the IR camera 13 while the subject 92 is placed at a predetermined location on the stage 90, and analyzing the positional shift (parallax ε) of the subject 92 acquired by each of the two cameras. The internal parameters and external parameters of the camera are estimated by the calibration. More specifically, the calibration is performed by having the subject 92 hold a calibration board with a lattice pattern, and associating the positions of each grid on the calibration board observed by the RGB camera 12 with the positions of each grid on the calibration board observed by the IR camera 13.

[0062] 8, it is assumed that calibration is performed with the subject 92 at position P0 (distance D between the camera and the subject 92=D0), and the parallax ε at this time is ε=ε0.

[0063] When the subject 92 moves from the position where the calibration was performed, the parallax ε differs from the state where the calibration was performed, and therefore accurate alignment between the RGB image acquired by the RGB camera 12 and the IR image acquired by the IR camera 13 cannot be performed.

[0064] For example, suppose that subject 92 approaches the camera from the position where calibration was performed and moves to position P1 (distance D between the camera and subject 92 = D1). At this time, the parallax ε = ε1 of subject 92 captured by RGB camera 12 and IR camera 13 becomes larger than the parallax ε = ε0 at position P0 where calibration was performed. In other words, ε1 > ε0. Therefore, even if information obtained by calibration is used, it becomes impossible to accurately extract the area of ​​subject 92.

[0065] Meanwhile, suppose that subject 92 moves away from the position where calibration was performed in a direction away from the camera to position P2 (distance D between the camera and subject 92 = D2). At this time, the parallax ε = ε2 of subject 92 captured by RGB camera 12 and IR camera 13 is smaller than the parallax ε = ε0 at position P0 where calibration was performed. In other words, ε2 < ε0. Therefore, even if information obtained by calibration is used, it becomes impossible to accurately extract the area of ​​subject 92.

[0066] The effect intensity calculation unit 35 sets the intensity of the image effect according to the distance (shooting distance) between the camera and the subject 92. Specifically, when the shooting distance is closer than a predetermined distance, that is, when the subject 92 is closer than the position where the calibration was performed (when the subject 92 is at a position where the parallax ε is larger than the position where the calibration was performed), an image effect with a higher intensity is applied than the intensity of the image effect applied when the subject 92 is at the position where the calibration was performed. This makes the position shift that occurs when the area of ​​the subject 92 is extracted less noticeable. Note that increasing the intensity of the image effect means applying a stronger image effect, such as significantly blurring the edges around the subject 92.

[0067] On the other hand, when the shooting distance is farther than the specified distance, i.e., when the subject 92 is farther away than the position where the calibration was performed (when the subject 92 is at a position where the parallax ε is smaller than the position where the calibration was performed), a video effect with the same strength or a weaker strength than the video effect applied when the subject 92 is at the position where the calibration was performed can be applied.

[0068] Here, the distance (shooting distance) between the camera and the subject 92 is calculated, for example, by the shooting distance calculation unit 34 based on the size (height) of the area of ​​the subject 92 extracted from the infrared image captured by the IR camera 13. Note that a distance measuring means such as a TOF (Time Of Flight) sensor may be installed near the RGB camera 12 or the IR camera 13, and the distance to the area of ​​the subject 92 may be measured by the distance measuring means.

[0069] [1-7. Flow of processing performed by the subject enhancement device] The flow of processing performed by the subject emphasizing device 20a will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the flow of processing performed by the subject emphasizing device of the first embodiment. Note that the flowchart in Fig. 9 shows the flow of processing after calibration is performed to obtain the positional relationship between the RGB camera 12 and the IR camera 13.

[0070] The visible light moving image acquisition unit 31 and the infrared moving image acquisition unit 32 simultaneously acquire an RGB image and an IR image while the IR light 14 is turned on (step S11).

[0071] The subject region extraction section 33 extracts the region of the subject 92 from the IR image acquired by the infrared video acquisition section 32 (step S12).

[0072] The shooting distance calculation unit 34 calculates the shooting distance from the IR camera 13 to the subject 92 (step S13).

[0073] The effect intensity calculation unit 35 calculates the intensity of the image effect based on the shooting distance calculated in step S13 (step S14).

[0074] The effect application section 36 applies a video effect to the area of ​​the subject 92 to generate an area of ​​the subject 93 on which the video effect has been applied (step S15).

[0075] The background drawing unit 37 generates and draws background information 94 (step S16).

[0076] The video composition unit 38 generates a composite video 95 by combining the background information 94 with the area of ​​the subject 93 on which the video effect has been applied (step S17).

[0077] The display control unit 39 causes the LED panel 15 to display the composite image 95 (step S18).

[0078] The visible light video capture unit 31 determines whether all images have been processed (step S19). If it is determined that all images have been processed (step S19: Yes), the subject highlighting device 20a ends the process in Fig. 9. On the other hand, if it is not determined that all images have been processed (step S19: No), the process returns to step S11.

[0079] [1-8. Effects of the first embodiment] As described above, in the subject emphasizing device 20a (information processing device) of the first embodiment, the visible light video acquisition section 31 (first acquisition section) acquires an RGB video (video) including the subject 92 and a background, and the subject region extraction section 33 extracts the region of the subject 92 from the video acquired by the visible light video acquisition section 31. Then, the background drawing section 37 draws background information 94 in a region other than the region of the subject 92 in the video acquired by the visible light video acquisition section 31. The effect application section 36 applies a video effect to the region of the subject 92 based on the intensity of the video effect to be applied to the region of the subject 92, which is determined by the effect intensity calculation section 35 according to the distance from the camera to the subject 92 (information related to the video acquisition environment). The video synthesis section 38 generates a synthetic video 95 by synthesizing the region of the subject 92 to which the video effect has been applied by the effect application section 36 and the background information 94 drawn by the background drawing section 37.

[0080] This makes it possible to extract the area of ​​the subject 92 under any background and apply video effects in real time. Also, the boundary between the subject 92 and the background information 94 can be expressed naturally.

[0081] Furthermore, according to the first embodiment of the subject emphasizing device 20a (information processing device), the effect application unit 36 ​​applies an effect to the area of ​​the subject 92 based on the intensity of the effect determined by the effect intensity calculation unit 35 in accordance with the shooting distance of the subject 92 calculated by the shooting distance calculation unit 34.

[0082] As a result, if there is a possibility that an error will occur in extracting the area of ​​the subject 92, a video effect can be applied that makes the error less noticeable.

[0083] Furthermore, according to the subject emphasizing device 20a (information processing device) of the first embodiment, when the shooting distance calculated by the shooting distance calculation unit 34 is closer than the shooting distance (predetermined distance) at the time of calibration, the effect application unit 36 ​​increases the intensity of the image effect and applies the image effect to the area of ​​the subject 92.

[0084] As a result, if there is a possibility that an error will occur in extracting the area of ​​the subject 92, a video effect can be applied that makes the error even less noticeable.

[0085] Furthermore, according to the subject emphasizing device 20a (information processing device) of the first embodiment, the infrared video acquisition section 32 (second acquisition section) acquires, in synchronization with the visible light video acquisition section 31 (first acquisition section), an intensity distribution (IR image) of reflected light of infrared light (invisible light) irradiated onto the subject 92, which is photographed at a predetermined relative position by the RGB camera 12. Then, the subject region extraction section 33 transforms the region of the subject 92 extracted from the IR image acquired by the infrared video acquisition section 32 into a shape and position expected to be acquired by the visible light video acquisition section 31, and fits it into the RGB image actually acquired by the visible light video acquisition section 31, thereby extracting the region of the subject 92.

[0086] This makes it possible to accurately extract the area of ​​the subject 92 with simple processing, even in a state in which the surrounding light environment changes drastically.

[0087] Moreover, according to the subject emphasizing device 20a (information processing device) of the first embodiment, the invisible light is infrared light.

[0088] This makes it possible to accurately extract the area of ​​the subject 92 through simple processing without being affected by the surrounding light environment.

[0089] Furthermore, in the subject emphasizing device 20a (information processing device) of the first embodiment, the display control section 39 (image display section) causes the composite image 95 composited by the image composition section to be displayed.

[0090] This makes it possible to combine the area of ​​the subject 93 to which the video effect has been applied with any background information 94.

[0091] Furthermore, according to the subject emphasizing device 20a (information processing device) of the first embodiment, the display control unit 39 (image display unit) displays the composite image 95 behind the subject 92.

[0092] This allows the viewer to view the subject 92 and the composite image 95 at the same time.

[0093] Furthermore, according to the subject emphasizing device 20a (information processing device) of the first embodiment, the effect application section 36 applies a video effect to the outer periphery of the subject 92 area.

[0094] This allows the subject 92 to be effectively emphasized.

[0095] [1-9. Modification of the first embodiment] Next, a description will be given of an object highlighting system 10b (not shown) which is a modified example of the first embodiment. The object highlighting system 10b includes an object highlighting device 20b instead of the object highlighting device 20a in the object highlighting system 10a described in the first embodiment.

[0096] FIG. 10 is a functional block diagram showing an example of a functional configuration of an object emphasizing device according to a modified example of the first embodiment.

[0097] The object emphasizing device 20b includes a disparity prediction unit 40 instead of the shooting distance calculation unit 34 included in the object emphasizing device 20a. The disparity prediction unit 40 predicts the maximum value of the deviation between the actual position of the object 92 and the extraction result of the object 92 based on the movement range of the object 92.

[0098] In the first embodiment, the subject emphasizing device 20a extracts the area of ​​the subject 92 based on the positional relationship between the RGB camera 12 and the IR camera 13, which is acquired in advance by calibration. The calibration is performed using a pair of an RGB image and an IR image that are captured simultaneously while the subject 92 is stationary at a predetermined position on the stage 90. Therefore, as described in the first embodiment, if the subject 92 moves on the stage 90, the area of ​​the subject 92 cannot be accurately extracted.

[0099] However, since the range of movement of the subject 92 is limited to the range of the stage 90, if the dimensions of the stage 90 are known in advance, the range of parallax ε between the subject 92 observed by the RGB camera 12 and the subject 92 observed by the IR camera 13 can be estimated.

[0100] In an object emphasizing device 20b which is a modification of the first embodiment, an effect intensity calculation section calculates the intensity of a video effect based on the maximum value of disparity ε of the object 92 predicted by the disparity prediction section .

[0101] More specifically, the effect intensity calculation unit 35 applies a video effect of an intensity according to the maximum value of parallax ε predicted to occur when the subject 92 moves around on the stage 90, regardless of the position of the subject 92.

[0102] That is, when the subject 92 is in a position where the parallax ε is greatest, by applying an image effect that makes the positional shift of the area of ​​the subject 92 extracted by the subject emphasizing device 20b less noticeable, the positional shift of the area of ​​the subject 92 can be made less noticeable wherever the subject 92 is on the stage 90.

[0103] Although a description of the flow of processing performed by the subject emphasizing device 20b will be omitted, in the flowchart of FIG. 9, instead of step S13, the disparity prediction unit 40 performs processing to calculate the maximum value of disparity ε based on information acquired by calibration and the dimensions of the stage 90. Then, in step S14, the effect intensity calculation unit 35 calculates the intensity of the video effect based on the maximum value of disparity ε. The rest of the processing flow is the same as that of FIG. 9. Also, these processes do not need to be performed for each image, but only once at the beginning.

[0104] [1-10. Effects of the Modification of the First Embodiment] As described above, in the subject emphasizing device 20b (information processing device) of the modified example of the first embodiment, the effect application unit 36 ​​applies a video effect to the area of ​​the subject 92 according to the maximum value of the disparity ε (deviation) between the actual position of the subject 92 and the extraction result of the subject 92, as predicted by the disparity prediction unit 40 based on the movement range of the subject 92.

[0105] This makes it possible to make the positional shift of the area of ​​subject 92 less noticeable even if subject 92 moves on stage 90. Also, since the intensity of the video effect only needs to be adjusted once when the first image is processed, it is possible to speed up processing.

[0106] 2. Second Embodiment The subject emphasis system 10a described in the first embodiment is equipped with two cameras (RGB camera 12 and IR camera 13), which causes a misalignment in the installation positions of the cameras and requires calibration. The subject emphasis system 10c (not shown) of the second embodiment is equipped with an RGB-IR camera 16 (see FIG. 11) that combines the RGB camera 12 and the IR camera 13 into a single camera.

[0107] The subject emphasis system 10c (not shown) of the second embodiment, as another example of a video acquisition environment, sets the intensity of a video effect based on the difference between the processing time of an IR image and the processing time of an RGB image, i.e., the processing delay time.

[0108] [2-1. Functional configuration of the subject enhancement device] The subject emphasizing system 10c of the second embodiment includes an RGB-IR camera 16, an IR light 14, an subject emphasizing device 20c, and an LED panel 15. That is, in FIG. 1, the RGB camera 12 and the IR camera 13 are integrated into the RGB-IR camera 16, and the subject emphasizing device 20a is replaced with the subject emphasizing device 20c.

[0109] First, the structure of the RGB-IR camera 16 will be described with reference to Fig. 11. Fig. 11 is a diagram illustrating the schematic structure of the RGB-IR camera.

[0110] FIG. 11 shows two types of structures of the RGB-IR camera 16. The RGB-IR camera 16a is a camera equipped with an image sensor 17 capable of simultaneously detecting visible light and infrared light. The image sensor 17 has a structure in which an element having sensitivity to infrared light is incorporated into a part of an image sensor having a Bayer array. That is, the image sensor 17 is equipped with a pixel IR having sensitivity to infrared light, in addition to pixels R, G, and B for generating an RGB image. Therefore, the RGB-IR camera 16a can acquire an RGB image and an IR image with a single camera. And since there is no parallax between the acquired RGB image and the IR image, it is possible to align the RGB image and the IR image without using the relative positional relationship of the cameras described in the first embodiment.

[0111] The RGB-IR camera 16b splits the optical path of the incident light into two by a beam splitter 18, and places a visible light imaging element 19a and an infrared imaging element 19b at the end of each optical path. The beam splitter 18 has a structure in which two right-angle prisms are bonded together.

[0112] Next, the functional configuration of the object emphasizing device 20c will be described with reference to Fig. 12. Fig. 12 is a functional block diagram showing an example of the functional configuration of the object emphasizing device of the second embodiment. That is, compared to the object emphasizing device 20a, the object emphasizing device 20c includes a video acquisition unit 41 instead of the visible light video acquisition unit 31 and the infrared video acquisition unit 32. Also, the object emphasizing device 20c includes a subject region extraction unit 42 instead of the object region extraction unit 33. Furthermore, the object emphasizing device 20c includes a processing time measurement unit 43 instead of the shooting distance calculation unit 34.

[0113] The video acquisition unit 41 acquires an RGB video (video) including the subject 92 and the background by acquiring a video signal captured by the RGB-IR camera 16. Furthermore, the video acquisition unit 41 acquires an IR video (video) including the subject 92 and the background by acquiring a video signal captured by the RGB-IR camera 16 at a timing when the IR light 14 is turned on. Note that the video acquisition unit 41 is an example of a first acquisition unit and a second acquisition unit in this disclosure.

[0114] The subject region extraction section 42 extracts the region of the subject 92 from the RGB video image acquired by the video acquisition section 41 .

[0115] The processing time measurement unit 43 measures the elapsed time since the moving image acquisition unit 41 (first acquisition unit) acquired the moving image.

[0116] [2-2. Description of subject area extraction process] Next, the contents of the subject region extraction process performed by the subject region extraction section 42 will be described with reference to Fig. 13. Fig. 13 is a diagram showing the flow of the subject region extraction process performed by the subject region extraction section of the second embodiment.

[0117] The RGB image and the IR image captured by the RGB-IR camera 16 are acquired by a moving image acquisition unit 41 .

[0118] 13, subject region extraction section 42 extracts the region of subject 92 from the RGB image acquired by moving image acquisition section 41. More specifically, subject region extraction section 42 extracts the region of subject 92 from the IR image acquired by moving image acquisition section 41, and extracts a region in the RGB image acquired by moving image acquisition section 41 that corresponds to the region of subject 92.

[0119] The subject region extraction unit 42 first calculates the difference (background difference 42a) between the IR image acquired by the video acquisition unit 41 and the background image captured by the RGB-IR camera 16 with the IR light 14 turned on and the subject 92 not being present. The region of the subject 92 has a higher infrared reflectance than the background image, and is therefore observed as bright in the IR image acquired by the video acquisition unit 41. Therefore, the background difference 42a results in an IR image in which the region of the subject 92 is brightly emphasized.

[0120] Next, the subject region extraction unit 42 performs threshold processing 42b on the IR image that has been subjected to background subtraction 42a. The threshold processing 42b is a process for extracting only pixels having pixel values ​​brighter than a preset threshold. The threshold processing 42b extracts a region (mask) corresponding to the region of the subject 92.

[0121] Since the region extracted by the threshold process 42b generally contains noise, the subject region extraction section 42 performs noise removal 42c. The noise removal 42c is performed, for example, by removing isolated regions whose area is equal to or smaller than a predetermined threshold.

[0122] Furthermore, the subject region extraction unit 42 performs a morphology process 42d. The morphology process 42d is a process performed on a binary image, for example, to fill holes that occur in the region. After that, in the first embodiment, the warping process 33e is performed, but in the present embodiment, no parallax occurs between the RGB image and the IR image, so the warping process is not necessary.

[0123] Meanwhile, the subject region extraction unit 42 performs a demosaic process 42e on the RGB image captured by the RGB-IR camera 16. The demosaic process 42e is, for example, a process of correcting the brightness of the image.

[0124] Next, the background drawing unit 37 draws background information 94 in the area other than the area (mask) of the subject 92 determined by the morphology processing 42d.

[0125] Then, image synthesis unit 38 synthesizes background information 94 drawn by background drawing unit 37 and the RGB image on which subject region extraction unit 42 has performed demosaic processing. As a result, a synthetic image 95 is generated in which the visible light image captured by RGB-IR camera 16, i.e., the image of subject 92, is synthesized at a position of background information 94 corresponding to the region of subject 92.

[0126] [2-3. Calculation of effect strength] Next, the calculation of the intensity of the video effect performed by the subject emphasizing device 20c will be described. When the subject region extracting unit 42 extracts the region of the subject 92, the processing performed on the IR image has a larger amount of processing than the processing performed on the RGB image. Therefore, the processing on the RGB image is completed earlier than the processing on the IR image. If the delay time of the time required for processing the IR image relative to the time required for processing the RGB image is within a predetermined time, the processing can be continued in real time (for example, video rate). However, if the time required for processing the IR image becomes long, it becomes impossible to continue the real-time processing, and a difference occurs between the region of the subject 92 extracted from the IR image and the actual region of the subject 92 in the RGB image.

[0127] When the delay time between the time required to process an IR image and the time required to process an RGB image exceeds a predetermined time, the subject emphasizing device 20c of this embodiment sets the intensity of the video effect to be strong, thereby making the position shift of the area of ​​the subject 92 less noticeable. More specifically, when the elapsed time measured by the processing time measuring unit 43 reaches a predetermined time, the effect intensity calculation unit 35 calculates the intensity of the video effect depending on whether or not the subject area extraction unit 42 has completed extraction of the area of ​​the subject 92.

[0128] That is, the processing time measurement unit 43 measures the time that has elapsed since the video acquisition unit 41 acquired the RGB image and the IR image, and applies a high-intensity video effect when the elapsed time exceeds a predetermined time. Note that the predetermined time is set to a time that allows the subject area extraction unit 42 to execute the process of extracting the area of ​​the subject 92 in real time (video rate).

[0129] [2-4. Processing flow of the subject enhancement device] The flow of processing performed by the subject emphasizing device 20c will be described with reference to Fig. 14. Fig. 14 is a flow chart showing an example of the flow of processing performed by the subject emphasizing device of the second embodiment.

[0130] The moving image acquisition unit 41 acquires an RGB image and an IR image (step S21).

[0131] The subject region extraction section 42 starts extracting the region of the subject 92 (step S22).

[0132] The processing time measurement unit 43 determines whether a predetermined time has passed since the image was acquired (step S23). If it is determined that the predetermined time has passed since the image was acquired (step S23: Yes), the process proceeds to step S24. On the other hand, if it is not determined that the predetermined time has passed since the image was acquired (step S23: No), the process proceeds to step S27.

[0133] If the result of the determination in step S23 is Yes, the subject region extraction unit 42 determines whether extraction of the region of the subject 92 is complete (step S24). If it is determined that extraction of the region of the subject 92 is complete (step S24: Yes), the process proceeds to step S28. On the other hand, if it is determined that extraction of the region of the subject 92 is not complete (step S24: No), the process proceeds to step S25.

[0134] On the other hand, if it is determined No in step S23, the subject region extraction unit 42 determines whether extraction of the region of the subject 92 is complete (step S27). If it is determined that extraction of the region of the subject 92 is complete (step S27: Yes), the process proceeds to step S28. On the other hand, if it is determined that extraction of the region of the subject 92 is not complete (step S27: No), the process returns to step S23.

[0135] Returning to step S24, if the result of the determination in step S24 is No, the subject region extraction unit 42 determines whether extraction of the region of the subject 92 is complete (step S25). If it is determined that extraction of the region of the subject 92 is complete (step S25: Yes), the process proceeds to step S26. On the other hand, if it is not determined that extraction of the region of the subject 92 is complete (step S25: No), step S25 is repeated.

[0136] If the determination in step S24 or step S27 is Yes, the effect intensity calculation section 35 sets the intensity of the video effect to a weak state (step S28), and then proceeds to step S29.

[0137] Also, if the result of the determination in step S25 is Yes, the effect intensity calculation section 35 sets the intensity of the video effect to strong (step S26), and then the process proceeds to step S29.

[0138] Following step S26 or step S28, the effect application section 36 applies a preset video effect to the area of ​​the subject 92 (step S29).

[0139] Next, the background drawing unit 37 generates and draws background information 94 (step S30).

[0140] The video composition unit 38 generates a composite video 95 by combining background information 94 with the area of ​​the subject 93 on which the video effect has been applied (step S31).

[0141] The display control unit 39 causes the LED panel 15 to display the composite image 95 (step S32).

[0142] The video acquisition unit 41 determines whether all images have been processed (step S33). If it is determined that all images have been processed (step S33: Yes), the subject highlighting device 20c ends the process in Fig. 14. On the other hand, if it is not determined that all images have been processed (step S33: No), the process returns to step S21.

[0143] [2-5. Effects of the second embodiment] As described above, according to the subject emphasizing device 20c (information processing device) of the second embodiment, the effect application unit 36 ​​applies a video effect to the area of ​​the subject 92 based on the intensity of the video effect determined depending on whether the subject area extraction unit 42 has completed extraction of the area of ​​the subject 92 when a predetermined time has elapsed since the video acquisition unit 41 (first acquisition unit) acquired the video, as measured by the processing time measurement unit 43.

[0144] As a result, even if the process of extracting the area of ​​the subject 92 takes a long time, the positional shift of the area of ​​the subject 92 in the composite image 95 can be made inconspicuous.

[0145] Furthermore, according to the subject emphasizing device 20c (information processing device) of the second embodiment, when the subject area extraction unit 42 has not completed extraction of the area of ​​the subject 92 when the elapsed time measured by the processing time measurement unit 43 has reached a predetermined time, the effect application unit 36 ​​increases the intensity of the video effect and applies the video effect to the area of ​​the subject 92.

[0146] As a result, if the process of extracting the area of ​​the subject 92 takes a long time, a strong image effect can be applied to make the positional shift of the area of ​​the subject 92 in the composite image 95 less noticeable.

[0147] [2-6. Modification of the second embodiment] Next, a description will be given of an object highlighting system 10d (not shown) which is a modification of the second embodiment. The object highlighting system 10d includes an object highlighting device 20d instead of the object highlighting device 20c described in the second embodiment.

[0148] The subject emphasizing system 10d controls the ON / OFF of video effects using a sudden change in the lighting environment as a trigger, such as shining a spotlight on the stage 90 or setting off fireworks on the stage 90.

[0149] Conventionally, ON / OFF of image effects was controlled based on a predetermined time table, which required troublesome prior setting. In contrast, the subject emphasis system 10d can turn image effects ON / OFF without having to set the ON / OFF times of the image effects in advance.

[0150] Specifically, when the number of pixels exceeding a predetermined brightness in the IR image acquired by the video acquisition unit 41 exceeds a predetermined value, the subject emphasizing device 20d determines that a sudden change in the lighting environment has occurred (a lot of infrared light has been irradiated), and turns the image effect ON or OFF.

[0151] Next, the functional configuration of the object emphasizing device 20d will be described with reference to Fig. 15. Fig. 15 is a functional block diagram showing an example of the functional configuration of an object emphasizing device according to a modified example of the second embodiment.

[0152] In addition to the functional components of the subject emphasizing device 20c, the subject emphasizing device 20d includes a trigger information detection unit 44 and an effect ON / OFF control unit 45. The trigger information detection unit 44 counts the number of pixels having brightness equal to or greater than a predetermined value in the IR image acquired by the video acquisition unit 41 (second acquisition unit).

[0153] The effect ON / OFF control unit 45 switches whether to apply a video effect, that is, whether to output the RGB image acquired by the video acquisition unit 41 as is, or to output the composite video 95 synthesized by the video synthesis unit 38, according to the number of pixels counted by the trigger information detection unit 44. The effect ON / OFF control unit 45 is an example of a switching unit in the present disclosure.

[0154] Next, the flow of processing performed by the subject emphasizing device 20d will be described with reference to Fig. 16. Fig. 16 is a flow chart showing an example of the flow of processing performed by the subject emphasizing device of the modified example of the second embodiment.

[0155] The moving image acquisition unit 41 acquires an RGB image and an IR image (step S41).

[0156] The subject region extraction section 42 extracts the region of the subject 92 (step S42).

[0157] The trigger information detection unit 44 judges whether the number of pixels having a predetermined brightness or more in the acquired IR image is equal to or greater than a predetermined value (step S43). If it is judged that the number of pixels having a predetermined brightness or more in the IR image is equal to or greater than the predetermined value (step S43: Yes), the process proceeds to step S44. On the other hand, if it is not judged that the number of pixels having a predetermined brightness or more in the IR image is equal to or greater than the predetermined value (step S43: No), the process proceeds to step S45.

[0158] If the result of the determination in step S43 is Yes, the effect ON / OFF control unit 45 stops the application of the video effect (step S44), and then the process proceeds to step S47.

[0159] On the other hand, if the result of the determination in step S43 is No, the effect intensity calculation section 35 sets the intensity of the video effect according to the delay time of the extraction process of the area of ​​the subject 92 (step S45).

[0160] Then, the effect application section 36 applies a video effect (step S46).

[0161] Next, the background drawing unit 37 generates and draws background information 94 (step S47).

[0162] The video composition unit 38 generates a composite video 95 by combining the background information 94 with the area of ​​the subject 93 on which the video effect has been applied (step S48).

[0163] The display control unit 39 causes the LED panel 15 to display the composite image 95 (step S49).

[0164] The video acquisition unit 41 determines whether all images have been processed (step S50). If it is determined that all images have been processed (step S50: Yes), the subject highlighting device 20d ends the process in Fig. 16. On the other hand, if it is not determined that all images have been processed (step S50: No), the process returns to step S41.

[0165] Although the above describes an example in which the illumination of a spotlight or the launch of fireworks is used as a trigger to turn off the video effects, the video effects may be turned on by these triggers instead. It is also possible to illuminate a spotlight using a projector capable of projecting a pattern of light, and switch the video effects on and off only in the area illuminated by the pattern.

[0166] The timing of spotlight illumination can also be determined according to the level of excitement at the live venue. For example, a sound level meter installed at the venue can measure sound pressure, and the spotlight can be illuminated when a certain level of sound pressure is reached. This allows the video effects to be turned on and off in response to the excitement at the venue, further enhancing the presentation effect.

[0167] Furthermore, the spotlight may be irradiated using a predetermined movement of the subject 92 as a trigger. The predetermined movement of the subject 92 can be detected, for example, by a method to be described later (third embodiment).

[0168] [2-7. Effects of the Modification of the Second Embodiment] As described above, in the subject emphasizing device 20d (information processing device) which is a modified example of the second embodiment, the effect ON / OFF control unit 45 (switching unit) switches whether or not to apply a video effect depending on the number of pixels having brightness equal to or greater than a predetermined value in the IR image acquired by the video acquisition unit 41 (second acquisition unit) counted by the trigger information detection unit 44.

[0169] This makes it possible to easily and reliably control the ON / OFF of a video effect without having to set in advance the time for which the effect is to be controlled ON / OFF.

[0170] 3. Third embodiment An object emphasizing system 10e (not shown) of the third embodiment detects the movement of an extracted object 92, and sets an area into which background information 94 is to be synthesized in accordance with the detected movement.

[0171] The subject emphasizing system 10e includes an object emphasizing device 20e instead of the object emphasizing device 20a in the subject emphasizing system 10a described in the first embodiment.

[0172] [3-1. Functional configuration of the subject enhancement device] First, the functional configuration of an object emphasizing device 20e will be described with reference to Fig. 17. Fig. 17 is a functional block diagram showing an example of the functional configuration of an object emphasizing device according to the third embodiment.

[0173] The object emphasizing device 20e has a functional configuration in which a motion detection section 46 and an object synthesis position setting section 47 are added to the object emphasizing device 20a (see FIG. 3) described in the first embodiment.

[0174] The motion detection unit 46 detects the motion of the subject 92. Specifically, the motion detection unit 46 detects the magnitude and direction of motion by detecting, for example, an optical flow (motion vector) for the areas of the subject 92 extracted by the subject area extraction unit 33 at different times.

[0175] Based on the movement of subject 92 detected by movement detection unit 46, subject synthesis position setting unit 47 sets a position where the area of ​​subject 93 to which the effect application unit 36 ​​has applied a video effect is to be synthesized with background information 94 drawn by background drawing unit 37. Specifically, subject synthesis position setting unit 47 sets the position of subject 92 to be synthesized with background information 94 (more precisely, the position of subject 93 to which the video effect has been applied) based on the magnitude and direction of the movement of subject 92. Note that subject synthesis position setting unit 47 is an example of a synthesis position setting unit in the present disclosure.

[0176] [3-2. Processing flow of the subject enhancement device] Next, the flow of processing performed by the subject emphasizing device 20e will be described with reference to Fig. 18. Fig. 18 is a flowchart showing an example of the flow of processing performed by the subject emphasizing device of the third embodiment. Note that the flowchart in Fig. 18 shows the flow of processing after calibration is performed to obtain the positional relationship between the RGB camera 12 and the IR camera 13.

[0177] The visible light moving image acquisition unit 31 and the infrared moving image acquisition unit 32 simultaneously acquire an RGB image and an IR image while the IR light 14 is turned on (step S51).

[0178] The subject region extraction section 33 extracts the region of the subject 92 from the IR image acquired by the infrared video acquisition section 32 (step S52).

[0179] The motion detection unit 46 detects the motion of the subject 92 (step S53).

[0180] The subject synthesis position setting unit 47 calculates an area in which the subject 92 is synthesized with the background information 94, that is, a background synthesis area (step S54).

[0181] Note that steps S55 to S61 are the same as the flow from steps S13 to S19 described in the first embodiment (see FIG. 9), and therefore a description thereof will be omitted.

[0182] [3-3. Effects of the third embodiment] As described above, in the subject emphasizing device 20e (information processing device) of the third embodiment, the subject synthesis position setting unit 47 (synthesis position setting unit) sets an area for synthesizing the area of ​​the subject 93 to which the effect application unit 36 ​​has applied a video effect, onto the background information 94 drawn by the background drawing unit 37, based on the movement of the subject 92 detected by the movement detection unit 46.

[0183] This makes it possible to determine, by simple processing, the position of the subject 92 to be combined with the background information 94 when the subject 92 is moving. In particular, the processing time required to detect the area of ​​the subject 92 can be increased.

[0184] Although the present disclosure has been described using several embodiments, these embodiments may be executed in any device as long as the device has the necessary functional blocks and is capable of obtaining the necessary information.

[0185] Also, for example, each step of one flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0186] Also, for example, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, so long as no contradiction occurs. Furthermore, the processing of the steps of a program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0187] Also, for example, a plurality of techniques related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any of a plurality of the present technologies can be applied and implemented. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-mentioned present technologies can be implemented in combination with other technologies not described above.

[0188] In addition, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may be achieved. In addition, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit and scope of the present disclosure.

[0189] For example, the present disclosure can also be configured as follows.

[0190] (1) a first acquisition unit that acquires a moving image including a subject and a background; a subject area extraction unit that extracts an area of ​​the subject from the moving image acquired by the first acquisition unit; a background drawing unit that draws background information in an area other than the area of ​​the subject in the moving image acquired by the first acquisition unit; an effect application unit that applies an effect to the area of ​​the subject based on the strength of the effect to be applied to the area of ​​the subject, the strength of the effect being determined according to information related to an acquisition environment of the video; an image synthesis unit that generates a synthetic image by synthesizing the area of ​​the subject to which the effect has been applied by the effect application unit and background information drawn by the background drawing unit; An information processing device comprising: (2) The effect application section includes: applying the effect to a region of the subject based on the strength of the effect determined according to a shooting distance of the subject; The information processing device according to (1). (3) The effect application section includes: When the shooting distance of the subject is closer than a predetermined distance, the intensity of the effect is increased and the effect is applied to the area of ​​the subject. The information processing device according to (2). (4) The effect application section includes: when a predetermined time has elapsed since the first acquisition unit acquired the video, the effect is applied to the area of ​​the subject based on the strength of the effect determined depending on whether or not the object area extraction unit has completed extraction of the area of ​​the subject. The information processing device according to any one of (1) to (3). (5) The effect application section includes: when the elapsed time has reached the predetermined time, if the subject region extraction unit has not completed extraction of the subject region, increasing the intensity of the effect and applying the effect to the subject region. The information processing device according to (4). (6) a second acquisition unit that acquires an intensity distribution of reflected light of invisible light irradiated on the subject, the reflected light being photographed at a predetermined relative position from the first acquisition unit in synchronization with the first acquisition unit, the subject region extraction unit transforms the subject region extracted from the intensity distribution of the reflected light acquired by the second acquisition unit into a shape and position expected to be acquired by the first acquisition unit, and extracts the subject region by fitting the transformed shape and position to the moving image actually acquired by the first acquisition unit. The information processing device according to any one of (1) to (5). (7) The invisible light is infrared light. The information processing device according to (6). (8) The image display unit may further include a video display unit that displays the composite image synthesized by the video synthesis unit. The information processing device according to any one of (1) to (7). (9) The image display unit displays the composite image behind the subject. The information processing device according to (8). (10) The effect application unit applies the effect to an outer periphery of the area of ​​the subject. The information processing device according to any one of (1) to (9). (11) The effect application section includes: applying the effect corresponding to the maximum value of the deviation between the actual position of the subject, which is predicted based on the movement range of the subject, and the extraction result of the subject, to the area of ​​the subject; The information processing device according to any one of (1) to (10). (12) A switching unit that switches whether or not to apply the effect depending on the number of pixels having brightness equal to or greater than a predetermined value in the intensity distribution of the reflected light acquired by the second acquisition unit, The information processing device according to (6) or (7). (13) a synthesis position setting unit that sets an area in which the area of ​​the subject to which the effect is applied by the effect application unit is to be synthesized with background information drawn by the background drawing unit based on the movement of the subject, The information processing device according to any one of (1) to (12). (14) A first acquisition step of acquiring a video including a subject and a background; a subject extraction step of extracting a region of the subject from the moving image acquired in the first acquisition step; a background drawing step of drawing background information in an area other than the area of ​​the subject in the moving image acquired in the first acquisition step; an effect application step of applying an effect to the area of ​​the subject based on the strength of the effect to be applied to the area of ​​the subject, the strength of the effect being determined according to information related to an acquisition environment of the video; an image compositing step of generating a composite image by combining the area of ​​the subject to which the effect has been applied in the effect application step with background information drawn in the background drawing step; A synthetic image generating method comprising: (15) Computer, a first acquisition unit that acquires a moving image including a subject and a background; a subject area extraction unit that extracts an area of ​​the subject from the moving image acquired by the first acquisition unit; a background drawing unit that draws background information in an area other than the area of ​​the subject in the moving image acquired by the first acquisition unit; an effect application unit that applies an effect to the area of ​​the subject based on the strength of the effect to be applied to the area of ​​the subject, the strength of the effect being determined according to information related to an acquisition environment of the video; an image synthesis unit that generates a synthetic image by synthesizing the area of ​​the subject to which the effect has been applied by the effect application unit and background information drawn by the background drawing unit; A program that makes it work. [Explanation of symbols]

[0191] Reference Signs List 10a, 10b, 10c, 10d, 10e...Subject enhancement system, 12...RGB camera (color camera), 13...IR camera (infrared camera), 14...IR light (infrared light), 15...LED panel, 16...RGB-IR camera, 20a, 20b, 20c, 20d, 20e...Subject enhancement device (information processing device), 31...Visible light video acquisition unit (first acquisition unit), 32...Infrared video acquisition unit (second acquisition unit), 33, 42...Subject area extraction unit, 34...Shooting distance calculation unit, 3 5...Effect intensity calculation unit, 36...Effect application unit, 37...Background drawing unit, 38...Video synthesis unit, 39...Display control unit (video display unit), 40...Disparity prediction unit, 41...Video acquisition unit (first acquisition unit, second acquisition unit), 43...Processing time measurement unit, 44...Trigger information detection unit, 45...Effect ON / OFF control unit (switching unit), 46...Motion detection unit, 47...Subject synthesis position setting unit (synthetic position setting unit), 90...Stage, 92, 93...Subject, 94...Background information, 95...Synthetic image, ε...Disparity

Claims

1. A first acquisition unit that acquires a moving image including a subject and a background; a subject area extraction unit that extracts an area of ​​the subject from the moving image acquired by the first acquisition unit; a background drawing unit that draws background information in an area other than the area of ​​the subject in the moving image acquired by the first acquisition unit; an effect application unit that applies an effect to the area of ​​the subject based on the strength of the effect to be applied to the area of ​​the subject, the strength of the effect being determined according to information related to an acquisition environment of the video; an image synthesis unit that generates a synthetic image by synthesizing the area of ​​the subject to which the effect has been applied by the effect application unit and background information drawn by the background drawing unit; Equipped with The effect application section includes: when a predetermined time has elapsed since the first acquisition unit acquired the video, the effect is applied to the area of ​​the subject based on the strength of the effect determined depending on whether or not the object area extraction unit has completed extraction of the area of ​​the subject; Information processing device.

2. The effect application section includes: applying the effect to a region of the subject based on the strength of the effect determined according to a shooting distance of the subject; The information processing device according to claim 1 .

3. The effect application section includes: When the shooting distance of the subject is closer than a predetermined distance, the intensity of the effect is increased and the effect is applied to the area of ​​the subject. The information processing device according to claim 2 .

4. The effect application section includes: when the elapsed time has reached the predetermined time, if the subject region extraction unit has not completed extraction of the subject region, increasing the intensity of the effect and applying the effect to the subject region. The information processing device according to claim 1 .

5. a second acquisition unit that acquires an intensity distribution of reflected light of invisible light irradiated onto the subject, the reflected light being photographed at a predetermined relative position from the first acquisition unit in synchronization with the first acquisition unit, the subject region extraction unit transforms the subject region extracted from the intensity distribution of the reflected light acquired by the second acquisition unit into a shape and position expected to be acquired by the first acquisition unit, and extracts the subject region by fitting the transformed shape and position to the moving image actually acquired by the first acquisition unit; The information processing device according to claim 1 .

6. The invisible light is infrared light. The information processing device according to claim 5 .

7. The image display unit may further include a video display unit that displays the composite image synthesized by the video synthesis unit. The information processing device according to claim 1 .

8. The image display unit displays the composite image behind the subject. The information processing device according to claim 7.

9. The effect application unit applies the effect to an outer periphery of the area of ​​the subject. The information processing device according to claim 1 .

10. The effect application section includes: applying the effect corresponding to the maximum value of the deviation between the actual position of the subject, which is predicted based on the movement range of the subject, and the extraction result of the subject, to the area of ​​the subject; The information processing device according to claim 1 .

11. a switching unit that switches whether or not to apply the effect depending on the number of pixels having brightness equal to or greater than a predetermined value in the intensity distribution of the reflected light acquired by the second acquisition unit, The information processing device according to claim 5 .

12. a synthesis position setting unit that sets an area in which the area of ​​the subject to which the effect is applied by the effect application unit is to be synthesized with background information drawn by the background drawing unit based on the movement of the subject, The information processing device according to claim 1 .

13. A first acquisition step of acquiring a moving image including a subject and a background; a subject extraction step of extracting a region of the subject from the moving image acquired in the first acquisition step; a background drawing step of drawing background information in an area other than the area of ​​the subject in the moving image acquired in the first acquisition step; an effect application step of applying an effect to the area of ​​the subject based on the strength of the effect to be applied to the area of ​​the subject, the strength of the effect being determined according to information related to an acquisition environment of the video; an image compositing step of generating a composite image by combining the area of ​​the subject to which the effect has been applied in the effect application step with background information drawn in the background drawing step; Including, The effect application step includes: applying the effect to the area of ​​the subject based on the strength of the effect determined depending on whether or not the area of ​​the subject has been extracted in the object extraction step when a predetermined time has elapsed since the first acquisition step acquired the moving image; A method for generating synthetic images.

14. Computer, A first acquisition unit that acquires a moving image including a subject and a background; a subject area extraction unit that extracts an area of ​​the subject from the moving image acquired by the first acquisition unit; a background drawing unit that draws background information in an area other than the area of ​​the subject in the moving image acquired by the first acquisition unit; an effect application unit that applies an effect to the area of ​​the subject based on the strength of the effect to be applied to the area of ​​the subject, the strength of the effect being determined according to information related to an acquisition environment of the video; an image synthesis unit that generates a synthetic image by synthesizing the area of ​​the subject to which the effect has been applied by the effect application unit and background information drawn by the background drawing unit; and make it work. The effect application section includes: when a predetermined time has elapsed since the first acquisition unit acquired the video, the effect is applied to the area of ​​the subject based on the strength of the effect determined depending on whether or not the object area extraction unit has completed extraction of the area of ​​the subject; program.

Images