Captured image selection device, captured image display device, and captured image selection program

Abstract

To efficiently image subjects in a volumetric capture studio when there are a plurality of performers.SOLUTION: A captured video selection device 1 selects one of a plurality of videos captured by a plurality of cameras surrounding one target subject in a volumetric capture studio as a captured video to be composited with a captured video of at least one separately recorded subject in the volumetric capture studio. The captured video selection device 1 includes a camera selection unit 41 for selecting a camera that is located at a position where the separately recorded subject does not obstruct the target subject during video composition and the front side of the target subject can be imaged.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to studio technology, and more particularly to a shot image selection device, a shot image display device, and a shot image selection program for presenting images of a subject shot by a plurality of imaging devices. [Background technology]

[0002] Conventionally, volumetric capture technology has been known. The volumetric capture technology is a technology for generating a three-dimensional model from images of a subject captured by multiple imaging devices (hereinafter, described as cameras). An observer of a three-dimensional model generated by this technology can view the three-dimensional model from any viewpoint by moving a virtual camera in a virtual three-dimensional space. The subject that is the source for generating such a three-dimensional model is often photographed in a dedicated studio. In the volumetric capture technology, information on the subject from various angles is required to obtain the shape and color information of the subject. For this reason, cameras are installed in the dedicated studio so as to surround the subject. In this dedicated studio, the subject performs, for example, an act, and the act is captured by multiple cameras. A three-dimensional model is created based on the obtained captured video of the subject (for example, an actor).

[0003] Patent Document 1 discloses a system that distributes free viewpoint video of a 3D model and assumes that multiple viewers will view it from their desired viewpoints. In a real-time viewing environment, when multiple images of a subject taken by multiple cameras from different directions are displayed and played on multiple display devices, the system disclosed in Patent Document 1 realizes a technology that notifies the camera taking the images according to the degree of attention based on the viewing state of the viewer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-163685 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to facilitate smooth filming in a volumetric capture studio, it is possible to send the filmed footage back to the subject, who then becomes an observer of the footage and uses it as a reference for his or her performance. In this case, the footage sent back needs to be easy for the performer to understand when performing. In addition, when multiple subjects are shot simultaneously in a studio, occlusion can occur, where multiple subjects overlap on the camera image, causing defects in the 3D model created based on this. Therefore, in volumetric capture studios, subjects are often shot one at a time to prevent defects from appearing in the 3D model. This makes it difficult to synchronize the timing of actions or eye contact when there are multiple performers. Even if it is possible to synchronize the performances of multiple performers, such shooting takes an enormous amount of time. The system disclosed in Patent Document 1 directly sends back images from a camera in a studio, so it is not possible to synchronize the performances of multiple performers when filming them one by one.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide a shot image selection device, a shot image display device, and a shot image selection program that efficiently shoot subjects in a volumetric capture studio when there are multiple performers. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the shot footage selection device of the present invention is a shot footage selection device that selects one of multiple shot footage shot by multiple cameras surrounding a single target subject in a volumetric capture studio as a shot footage to be combined with shot footage of at least one separately recorded subject recorded separately in the volumetric capture studio, and is equipped with a camera selection unit that selects a camera that is positioned so that the separately recorded subject does not obstruct the target subject during video combination and is positioned so that it can capture the front side of the target subject.

[0008] According to this configuration, the shot video selection device includes a camera selection unit that selects an optimal camera from among a plurality of cameras that acquire a plurality of shot images of a subject from various directions in a volumetric capture studio. The camera selection unit selects a camera that is located in a position where the subject recorded separately does not block the subject in the studio during video synthesis. Therefore, a synthesized video using the camera video can be provided as a return video that enables a co-starring with a subject recorded separately. The subject who sees the return video can confirm his / her own movements and the movements of the subject recorded separately from the video, and can use the video as a reference for performing in accordance with the movements of the subject recorded separately. In addition, the camera selection unit selects a camera that is located in a position that can shoot the front side of the subject in the studio, so that the synthesized video using the camera video is a video of the front side of the subject in the studio, which is easier to understand when performing compared to a video of the back side of the subject in the studio. Therefore, the shot video selection device can smoothly proceed with shooting of the subject in the studio. The captured image selection device can be operated by a captured image selection program that causes a computer to function as the camera selection unit described above.

[0009] In addition, the captured image display device of the present invention comprises the captured image selection device, an imaging device having a plurality of cameras including the posture estimation camera, a recording device that records the separately recorded pre-recorded image of the subject and three-dimensional position information for each part of the separately recorded subject, a posture information acquisition device that acquires the three-dimensional position information for each part of the subject as posture information of the subject from the video signal of the posture estimation camera and projects it onto the image coordinate system of each camera, a video receiving device that receives and records multiple videos of the target subject captured by the plurality of cameras, and a display device that combines the pre-recorded separately recorded image of the subject with the image of the target subject captured by the camera selected by the camera selection unit, and displays the image on a display unit with marks on parts outside the area where the 3D model can be acquired.

[0010] According to this configuration, the captured image display device displays a return image that is a composite of separately recorded images of the subject based on the image of the subject captured by the camera of the imaging device and the posture information of the subject shown therein, making it possible to create a 3D model with synchronized movement timing for multiple people even when shooting one person at a time. Also, the captured image display device marks parts outside the area where a 3D model can be obtained in the return image that is a composite of separately recorded images of the subject, so that it is possible to call attention to the subject not to go out of the frame of the camera in the studio during shooting. Effect of the Invention

[0011] The present invention provides the following excellent effects. The shot image selection device according to the present invention selects a camera that captures images that are easy for a subject in a studio to match the movements of a subject recorded separately and that are easy to understand when performing. Therefore, the shot image selection device can efficiently shoot subjects in a studio when there are multiple performers by using a composite image using the selected camera images as a return image that enables the subject to perform together with the subject recorded separately. In addition, the captured image display device according to the present invention can display a return image that is a composite of an image of a subject recorded separately, so that when producing a work that includes a scene in which multiple performers perform related actions, it becomes easier to synchronize the timing of the actions. Furthermore, the captured image display device can call attention to the subject not to go out of the frame of the camera in the studio during shooting, so that defects can be prevented from occurring in the three-dimensional model to be created. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram showing a configuration of a shot video selection device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of camera placement in a studio. [Diagram 3] 11A and 11B are diagrams illustrating an example of joint points used as posture information. [Figure 4] FIG. 2 is a diagram showing an example of an area in a studio where a subject is blocked. [Diagram 5] FIG. 13 is a diagram showing an example of the body orientation of a subject in a studio. [Figure 6] FIG. 13 is a schematic diagram showing a configuration of a captured video selection device according to a modified example of the first embodiment of the present invention. [Figure 7] FIG. 1 is a schematic diagram showing a flow of video synthesis. [Figure 8] FIG. 11 is a schematic diagram showing a configuration of a shot video selection device according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a schematic diagram showing a configuration of a captured video selection device according to a modified example of the second embodiment of the present invention. [Figure 10] 13 is an example of an image showing that there is a part outside the area where a 3D model can be obtained. [Figure 11] FIG. 11 is a schematic diagram showing a configuration of a captured image display device according to a third embodiment of the present invention. [Figure 12] 12 is a flowchart showing a process flow of the captured image display device of FIG. 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the shot image selection device and the shot image display device according to the present invention will be described in order with reference to the drawings. Note that the size and positional relationship of the components shown in each drawing may be exaggerated for clarity of explanation. In addition, hereinafter, the volumetric capture studio may be simply called the studio.

[0014] (First embodiment) [1. Overview of the configuration of the shot video selection device according to the first embodiment] The configuration of a shot video selection device according to an embodiment of the present invention will be described with reference to Figures 1 and 2. The shot video selection device 1 performs a calculation process to select one of a plurality of shot videos captured by a plurality of cameras surrounding one target subject in a volumetric capture studio as a shot video to be composited with a shot video of at least one subject separately recorded in the volumetric capture studio. The captured image selection device 1 includes a camera selection unit 41. The camera selection unit 41 selects a camera that is positioned so that a separately recorded subject does not obstruct the target subject during image synthesis and that is positioned so that the front side of the target subject can be captured.

[0015] As shown in FIG. 1, the camera selection unit 41 includes a primary selection unit 100 and a secondary selection unit 200. The primary selection unit 100 uses posture information of the target subject and the separately recorded subject and position information of a plurality of cameras to exclude cameras whose circles representing the target subject as seen from the camera are in the shadow of the circles representing the separately recorded subject, based on the positions of circles projected onto a plane corresponding to the floor surface of the studio and the positions of each camera on the plane, and selects the remaining cameras. Here, the posture information is three-dimensional position information estimated by a posture estimation camera for each part of the body of each of the target subject and the separately recorded subject. Details of each functional unit of the primary selection unit 100 will be described later.

[0016] In this embodiment, the secondary selection section 200 includes a subject angle calculation section 210 , a difference index calculation section 220 , and a priority calculation section 230 . The subject angle calculation unit 210 uses the posture information of the target subject and the separately recorded subject to calculate a subject angle θ , which indicates the front direction of each subject with respect to a reference line that is predetermined on a plane onto which each subject is projected. h (see FIG. 2(b)). In this embodiment, the reference line is set to the Z axis (see FIG. 2(b)) as an example.

[0017] The difference index calculation unit 220 calculates the camera angle θ ci (See Fig. 2(b)) and the subject angle θ h The smaller the difference between the camera and the camera angle θ, the smaller the camera's position is to capture the front side of the subject. ci and the subject angle θ h The difference between these is calculated for each subject. The priority calculation unit 230 calculates a priority for each camera defined by weighting and adding, for the same camera, a first index indicating the difference between the camera angle and the subject angle for the target subject, and a second index indicating the difference between the camera angle and the subject angle for a subject recorded separately, and selects the camera with the smallest priority from among the cameras selected by the primary selection unit 100.

[0018] In the captured image selection device 1, the camera selection unit 41 selects a camera, which means selecting an image of a subject captured by that camera (camera image). The separately recorded subject is a subject captured separately from the target subject in a studio, and the separately recorded subject is also called a separate shot. In this embodiment, the separately recorded subject and the target subject are captured in the same studio, for example. The position and orientation of the camera capturing the separately recorded subject are the same as the position and orientation of the camera capturing the target subject, and the camera parameters are fixed. The separately recorded subject is usually different from the target subject, but in some cases it is possible to make them the same.

[0019] Various information required for the computation processing in the camera selection unit 41 is input to the camera selection unit 41 from an external device (not shown). Such information includes posture information of the target subject, posture information of subjects recorded separately, etc. Note that the position information of each camera in the studio may be stored in advance in the device. In the following, in order to explain each functional part of the camera selection unit 41, the arrangement and attitude information of the cameras will be described in order with reference to the drawings.

[0020] [2. Camera placement example] An example of the imaging device 10 in a volumetric capture studio will be described with reference to Fig. 2. In Fig. 2, the height direction is the Y axis, the optical axis of the posture estimation main camera (posture estimation camera) 11a is the Z axis, the optical axis of the posture estimation sub camera (posture estimation camera) 11b is the X axis, and the intersection of the two optical axes perpendicularly to the floor surface is the origin. Fig. 2(a) is a schematic diagram corresponding to one cross section parallel to the XY plane when the studio is viewed from the front. Fig. 2(b) is a schematic diagram corresponding to one cross section parallel to the XZ plane when the studio is viewed from above. Cameras at different heights are arranged in the studio to surround a subject Ha.

[0021] Here, the photographing device 10 is equipped with two dedicated posture estimation cameras (a posture estimation main camera 11a and a posture estimation sub-camera 11b) in order to estimate the posture of the subject. In the following, the cameras that can be used to send back images from within the studio are camera C. i Or we simply write it as camera, where i is the camera number (i=1, 2, …, N). Camera C i The position information of is expressed as three-dimensional coordinates (C ix ,C iy ,C iz ) is set based on the orientation of the main camera 11a for posture estimation. i The horizontal inclination of is θ ci The vertical inclination is φ i Let us assume that. The shape of the studio may be a cylinder as shown in FIG. 2, or may be a rectangular parallelepiped or hemispherical shape.

[0022] [3. Examples of posture information] The posture information means three-dimensional position information of the joint points of the subject. In this embodiment, it is assumed that the subject is a person. FIG. 3 is a diagram showing an example of the joint points used in the posture information. FIG. 3 shows a total of 19 parts, such as the head, neck, right shoulder, and left shoulder, as parts that characterize the human body. Note that all parts for specifying the posture, including parts that are not joints themselves, are called joint points. In addition, the joint points may include parts such as the right ear, left ear, right eye, and left eye as parts that characterize the human face. Here, the coordinates of the joint points are three-dimensional coordinates in the world coordinate system set as shown in Fig. 2(a) and Fig. 2(b). Hereinafter, the three-dimensional coordinate points (three-dimensional position information) of each part j of the subject are referred to as points P j (P jx ,P jy ,P jz ), where j indicates identification information of the subject part (j=1, 2, ..., M). Note that M is the total number of parts.

[0023] The pose information can be acquired by performing pose estimation, which can use a known pose estimation algorithm capable of detecting joint points of a person. As a posture estimation algorithm, for example, OpenPose or VisionPose (registered trademark of Next System Co., Ltd.) can be used. OpenPose is described in Reference 1 below.

[0024] Reference 1: Zhe Cao et al., “OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields”, arXiv preprint arXiv:1812.08008(2018)

[0025] In this embodiment, VisionPose is used to acquire 3D coordinate information of the subject. VisionPose can acquire 3D coordinate information of the subject's joint points based on 2D coordinate information of the subject's joint points acquired from two camera images or videos by inputting the arrangement information of the two cameras used for posture estimation, without using camera parameter estimation or a stereo camera method (epipolar geometry method).

[0026] [4. Details of Camera Selection Section 41] [4-1. Primary selection section 100] First, a method for determining a camera located at a position where a separately shot (separately recorded) subject does not obstruct the target subject by the primary selection section 100 of the camera selection section 41 will be described with reference to FIG. 4 (and also with reference to FIG. 1 as necessary). FIG. 4(a) is a schematic diagram of the target subject Ha and the separately shot subject Hb viewed from above the studio. FIG. 4(b) is a schematic diagram of the camera's shooting area viewed from above. In this embodiment, as shown in FIG. 1, the primary selection section 100 has a subject figure calculation section 110, a camera relative position calculation section 120, and an excluded camera determination section 130. Each section will be described in detail below.

[0027] [Subject figure calculation unit 110] The object figure calculation unit 110 calculates the center coordinates and the radius of each circle representing the target object and the separately recorded object on the plane onto which each object is projected. The calculation results of the object figure calculation unit 110 are output to the camera relative position calculation unit 120. The subject figure calculation unit 110 obtains information such as the position of the reference joint point from the posture information for each subject. Here, as an example, the reference joint points are the right shoulder and the left shoulder. This is done by dividing the circle S into two parts, with the line segment connecting the coordinates of both shoulders of each subject Ha and Hb as the diameter when the subjects Ha and Hb are viewed from above the studio as shown in FIG. 4(a). a ,S b The subject figure calculation unit 110 determines the coordinates of the right shoulder (x r ,z r ) and the coordinates of the left shoulder (x l ,zl Next, the subject figure calculation unit 110 obtains the coordinates of the midpoint between the coordinates of the right shoulder and the coordinates of the left shoulder ((x r +x l ) / 2,(z r +z l ) / 2). In addition, the distance from the midpoint to the coordinates of the right shoulder is obtained as the radius of the assumed circle. Note that the distance (radius) is expressed as the following formula (1).

[0028]

number

[0029] [Camera relative position calculation unit 120] The camera relative position calculation unit 120 calculates, for each camera, the slope of the tangent line from the camera to each circle and the distance from the camera to the center of each circle, using the position information of the multiple cameras and the calculation results by the subject figure calculation unit 110. The calculation results by the camera relative position calculation unit 120 are output to the excluded camera determination unit 130.

[0030] Here, camera C i Coordinates of (C ix ,C iz ) indicates, for example, the center point of a pixel of an image sensor. The camera relative position calculation unit 120 calculates the circle S b and the center coordinates and radius of the i-th camera C i The coordinate point (C ix ,C iz ) and the coordinate point (C ix ,C iz ) to yen S b Two tangent lines T that are tangent to b1 ,T b2 The slope m b1 ,m b2 (m b1 >m b2 Similarly, the camera relative position calculation unit 120 calculates the circle S a The center coordinates and radius of the coordinate point (C ix ,Ciz ) and the coordinate point (C ix ,C iz ) to yen S a Two tangent lines T that are tangent to a1 ,T a2 The slope m a1 ,m a2 (m a1 >m a2 ) are calculated respectively. Next, the camera relative position calculation unit 120 calculates the point (C ix ,C iz ) to yen S a Distance d to the center coordinate of a And point (C ix ,C iz ) to yen S b Distance d to the center coordinate of b And asks for.

[0031] [Exclusion camera determination unit 130] The excluded camera determination unit 130 excludes cameras for which the slopes of the two tangents to the circle representing the target subject are determined to be greater than the slope of one tangent to the circle representing the separately recorded subject and less than the slope of the other tangent, and selects the remaining cameras. Specifically, the excluded camera determination unit 130 calculates each tangent T a1 ,T a2 ,T b1 ,T b2 The slope m a1 ,m a2 ,m b1 ,m b2 About m b1 >m a1 >m b2 And m b1 >m a1 >m b2 The above conditions are met, and camera C i For the distance from the camera to the subject, d a >d bThe camera that satisfies the above conditions is extracted. The video from the camera that is extracted by satisfying these conditions is assumed to be a video in which the separately shot subject Hb obstructs the in-studio subject Ha (a video in which the in-studio subject Ha overlaps the separately shot subject Hb). The excluded camera determination unit 130 selects cameras that do not satisfy these conditions. In other words, the excluded camera determination unit 130 selects cameras in which the separately shot subject does not overlap the in-studio subject. The result selected by the excluded camera determination unit 130 is output to the secondary selection unit 200. Note that, although the case where the separately shot subject is one person has been described here, the primary selection unit 100 of the camera selection unit 41 similarly determines whether or not the subjects in the studio overlap with all of the separately shot subjects when there are two or more people.

[0032] [4-2.Second selection section 200] Next, a determination method for selecting a camera that shoots a subject from as far in front as possible by the secondary selection unit 200 of the camera selection unit 41 will be described with reference to Fig. 5(a) (and Figs. 1 to 4 as appropriate). Fig. 5 is a diagram showing an example of the orientation of the subject's body in a studio. As described above, the secondary selection section 200 includes the subject angle calculation section 210, the difference index calculation section 220, and the priority calculation section 230. Each section will be described in detail below.

[0033] [Subject angle calculation unit 210] The subject angle calculation unit 210 uses the posture information of the subject to calculate the front direction of the subject (subject angle θ h 2(b)). In this embodiment, the front direction of the subject is defined as the direction of the subject's body, for example. For this purpose, subject angle calculation section 210 calculates the direction of the body using the coordinates of the subject's right shoulder and left shoulder, for example. Here, the front direction of the studio is the direction in which the posture estimation main camera 11a (see FIG. 2(b)) is viewed from the center of the studio. In other words, the negative direction of the Z axis is the front direction of the studio. In addition, when a subject faces the posture estimation main camera 11a directly, the subject angle θ h is 0 degrees. Object angle θh The magnitude of increases clockwise from the third quadrant to the second quadrant to the first quadrant to the fourth quadrant, and 0≦θ h <=360.

[0034] As described above, the subject figure calculation unit 110 of the primary selection unit 100 selects the coordinates of the right shoulder on the XZ plane (x r ,z r ) and the coordinates of the left shoulder (x l ,z l Similarly, the subject angle calculation unit 210 of the secondary selection unit 200 also obtains the coordinates of the right shoulder (x r ,z r ) and the coordinates of the left shoulder (x l ,z l In Figure 5(a), the z coordinate of the right shoulder minus the z coordinate of the left shoulder is 0 or more (z l ≦z r ) so that the subject angle θ h It is assumed that:

[0035] Here, the subject angle calculation section 210 includes a straight line calculation section 212 and an angle calculation section 214, as shown in FIG. The straight line calculation unit 212 uses the posture information of each subject to calculate, for each target subject and separately recorded subject, a straight line α (see Figure 5(a); first straight line) which is the diameter of a circle representing the subject on a plane onto which the subject is projected and indicates the width direction of the subject, and a straight line β (see Figure 5(a); second straight line) which passes through the center of the circle and is perpendicular to the straight line α.

[0036] The line α shown in FIG. 5(a) is the coordinate of the subject's right shoulder (x r ,z r ) and the coordinates of the left shoulder (x l ,z l The line calculation unit 212 calculates the gradient of the line α. Here, the gradient of the line α is m c In addition, the line calculation unit 212 calculates the gradient of a line β perpendicular to the line α. Here, the gradient of the line β is m d Let (m d =-1 / m c) Note that the line L shown in FIG. 5(a) is a line parallel to the Z axis. The slope m of the straight line β calculated by the straight line calculation unit 212 d is output to the angle calculation unit 214.

[0037] The angle calculation unit 214 calculates the angle between the reference line (Z axis) and the line β (second line) for each subject, and calculates the subject angle θ based on the calculated angle and the posture information of each subject. h The direction of the subject in the studio (subject angle θ h ) indicates how much the object is tilted from the reference line (Z axis). In other words, the object angle θ h is the slope m d This shows how much the straight line β is inclined from the reference line (Z axis). Therefore, the angle calculation unit 214 calculates the Z axis and the inclination m d Find the angle with the line β.

[0038] Here, the Z axis and the slope are m d The angle between the line β and the Z axis can be expressed by using the inner product of a vector on the Z axis and a vector on the line β. For example, assume a direction vector d1 on the Z axis as shown in FIG. 5(a). Also, the slope m d Let us assume a vector on the line β of the vector, and assume a directional vector d2 after translating it so that the starting point of the vector coincides with the origin. In the XZ plane, the components of the directional vector d1 can be expressed as (0,1), for example. The components of the directional vector d2 can be expressed as (1,m d The angle θ between the direction vectors d1 and d2 can be expressed as the following formula (2) using the magnitude of these direction vectors d1 and d2 and the inner product of the vectors. Note that 0≦θ≦180.

[0039]

number

[0040] The angle calculation unit 214 calculates the angle θ between the reference line (Z axis) and the straight line β (second straight line) for each subject using equation (2). l ≦z r If the object angle θ h corresponds to the angle θ between the direction vectors d1 and d2 (θ h = θ). In other words, if the angle θ between the Z axis and the line β is, for example, 30 degrees, then the subject angle θ h is also 30 degrees.

[0041] Also, as shown in FIG. 5(b), if you subtract the z coordinate of the left shoulder from the z coordinate of the right shoulder, the result is negative (z l >z r ), the angle calculation unit 214 also calculates the Z axis and the inclination angle m d In this case, the angle between the object and the line β is calculated. h does not coincide with the angle θ between the direction vectors d1 and d2, but with the value obtained by adding 180 degrees to the angle θ (θ h =θ+180). In other words, if the angle θ between the Z axis and the line β is, for example, 150 degrees, then the subject angle θ h is 330 degrees.

[0042] In this way, the angle calculation unit 214 takes into consideration the z coordinates of the right shoulder and the z coordinates of the left shoulder, and calculates either the angle θ itself between the Z axis and the straight line β, or a value obtained by adding 180 degrees to the angle θ, as the front direction of the subject in the studio (subject angle θ h The object angle θ calculated by the angle calculation unit 214 is calculated as follows. h is output to the differential indicator calculation unit 220.

[0043] [Difference index calculation unit 220] The difference index calculation unit 220 calculates the camera angle θ ci (See Fig. 2(b)) and the subject angle θ h The difference between the camera angle θ ciindicates the horizontal tilt of the i-th camera, and is the rotation angle when the front direction of the main camera 11a for posture estimation is set to 0 degrees.

[0044] The difference index calculation unit 220 will be described with reference to FIG. 5(a) (and also with reference to FIGS. 1 to 4 as appropriate). In the example shown in FIG. 5(a), the angle at which each camera views the center of the studio with the positive direction of the Z axis as the reference (camera angle θ ci ) increases clockwise for each camera, and 0≦θ ci ≦360. In this example, the camera angle θ ci The camera with a 30 degree angle is the object (θ h It can be seen that this is a camera that takes pictures as close to the front as possible (for example, 30 degrees).

[0045] In this embodiment, the differential indicator calculation unit 220 calculates |θ h -θ ci | and (360-|θ h -θ ci |) is the smaller of these two, which is the index θ di The index θ di The smaller the difference value, the more the camera is positioned to capture the front side of the subject. di <=180.

[0046] θ di =min(|θ h -θ ci |,360-|θ h -θ ci |) … Formula (3)

[0047] The reason why the differential index calculation unit 220 uses the formula (3) is as follows. di is roughly the subject angle θ h and each camera angle θ ci The absolute value of the difference between h -θ ci As a concrete example, the camera angle θ ci is 30 degrees, and the subject angle θ hAssume that the absolute value of the difference between them (|θ h -θ ci |) is 300 degrees. However, if we consider it counterclockwise on the XZ plane, the effective angle difference is 60 degrees. Equation (3) takes into account such a case.

[0048] The difference index calculation unit 220 calculates the subject angle θ h For each camera, the index θ di Similarly, using equation (3), the difference index calculation unit 220 calculates the subject angle θ h For each camera, the index θ di (2nd index) is calculated. In the following, when distinguishing between indices, the 1st index θ di , second index θ di The first index θ′ calculated by the differential index calculation unit 220 is expressed as di and the second index θ di ' is output to the priority calculation unit 230. In this embodiment, the subject's body orientation is taken into consideration by using the coordinates of the subject's shoulders, but as a variant, the subject's face orientation can also be used by using the coordinates of the subject's eyes and ears.

[0049] [Priority calculation unit 230] The priority calculation unit 230 calculates the first index θ di and the second index θ di Based on this, the camera priority R is calculated by the following formula (4): i Calculate.

[0050] R i = aθ di +bθ di '...Equation (4)

[0051] Here, camera C i Priority R for i is the first index θ di and the second index θ diIt is defined as the weighted addition of a and b for the same camera (same camera number i). As mentioned above, the smaller the index value for each camera, the more the camera is positioned to capture the front side of the subject, so the priority calculated by weighting and adding them also naturally indicates that the smaller the value, the more the camera is positioned to capture the front side of the subject. Also, in equation (4), a and b are the weights of each parameter. For example, a=3 and b=1 are set. Since it is usually the subject in the studio who views the returned video, a>b is set with importance placed on the direction of the subject's body in the studio. Priority calculation unit 230 calculates the camera priority R i The identification information (selected camera information) of the camera with the smallest value is output. This selected camera information is used as the camera information to be used for the returned video. Note that even if there are multiple subjects in the separately shot video, the camera priority is calculated in the same way to determine the selected camera information.

[0052] As described above, the shot image selection device 1 selects a camera from which the camera image seen by the performer in the studio is taken from a position that shows as much in front of the performer as possible while taking into consideration the positional relationship with the performer shot separately. The image that the performer in the studio wants to see is an image that can be used as a reference for the performance by viewing it. Therefore, even if an image from a camera position behind the performer in the studio or an image from a camera position where the performer in the studio is blocked by the performer shot separately and cannot be seen, it will not be a reference for the performance of the performer in the studio even if it is sent back to the studio. For the performer in the studio, a camera image from a position that shows as much in front of the performer as possible is an appropriate image for checking his / her own performance. Also, by viewing an image from a camera position where the performer in the studio can see himself / herself without being blocked by the performer shot separately, the performer in the studio can perform while matching his / her movements with the performer shot separately. In this way, by selecting an appropriate camera that provides a camera image that is easy for the performer in the studio to act, the studio shooting can proceed smoothly.

[0053] 5. Modifications of the captured video selection device according to the first embodiment 6, the captured video selection device 1A according to the modified example includes an image processing unit 6. An operation instruction signal including selected camera information determined by a camera selection unit 41 is input to the image processing unit 6. Based on the operation instruction signal, the image processing unit 6 performs image processing using a video signal input from a camera selected by the camera selection unit 41 to generate a return video.

[0054] Studio equipment (camera, recorder, monitor, etc.) not shown is connected to the image processing unit 6. For example, a video signal of a target subject is input to the image processing unit 6 from an imaging device (camera). Also, a video signal of a subject recorded separately is input to the image processing unit 6 from a video recording device (recorder). Also, the image processing unit 6 outputs the sent back video to a display device (monitor).

[0055] As shown in Fig. 6, the image processing unit 6 includes a subject extraction unit 61 and a video synthesis unit 62. The subject extraction unit 61 extracts a subject from a separately recorded video of a subject corresponding to the camera selected by the camera selection unit 41, and also extracts a subject from a camera video corresponding to the camera selected by the camera selection unit 41. The separately recorded video of the subject is recorded in advance in, for example, a video recording device (recorder). Here, a semantic segmentation method, for example, DeepLab v3, is used to extract the subject. Note that DeepLab v3 is described in Reference 2 below.

[0056] Reference 2: Liang-Chieh Chen, George. Papandreou, Florian. Schroff, Hartwig. Adam. “Rethinking Atrous Convolution for Semantic Image Segmentation”, European Conference on Computer Vision (ECCV), 2017

[0057] In this embodiment, the subject extraction unit 61 extracts a person part from the video by DeepLab v3, and extracts pixels corresponding to the area determined to be the person part as the subject. The subject extraction unit 61 extracts a subject from a separately shot video (pre-recorded camera image) that is recorded in advance for the video from the camera selected by the camera selection unit 41, and also extracts a subject from the video from the camera in the studio.

[0058] The video synthesis unit 62 synthesizes and creates an image to be displayed as a return video using the subject extraction results for the separately recorded video of the subject and the video of the target subject. The video synthesis unit 62 synthesizes the separately shot subject image with the in-studio camera image using the extraction results from the subject extraction unit 61. The video synthesis unit 62 synthesizes the video taking into consideration the context between the performers in the studio and the separately shot performers.

[0059] The flow of image synthesis by the image synthesizing unit 62 will be described with reference to FIG. First, the video synthesis unit 62 deletes the part extracted as the subject from the in-studio camera image 101 and creates an image I1 by extracting the background other than the subject. Here, the in-studio camera image 101 is an image taken by a camera selected by the camera selection unit 41, for example.

[0060] Next, the video synthesis unit 62 creates an image I2 by pasting a portion 202 extracted as a subject from the recorded camera image 201 onto the image I1. Here, the recorded camera image 201 is an image taken by a camera facing in the same direction as the camera selected by the camera selection unit 41, for example.

[0061] Finally, the video synthesis unit 62 creates an image I3 by pasting a portion 301 extracted as a subject in the camera image (in-studio camera image 101) selected by the camera selection unit 41 onto the image I2. Note that image I1, which is the background of the synthesized image, is not limited to an in-studio camera image, and a separately prepared real-life image or CG image of the same image size may be used. Also, in FIG. 7, a scene in which the presenter makes a gesture to guide the guest forward is synthesized, but it is also possible to synthesize a scene in which multiple performers perform related actions such as shaking hands, high-fiving, and dancing.

[0062] The shot video selection device 1A according to the modified example creates a composite video by considering the positional relationship between the performers in the studio and the performers shot separately. Therefore, by providing the performers in the studio with a composite video that makes it easy for them to act, the shooting time required to combine the performances of multiple performers can be reduced, and studio shooting can be carried out efficiently. Also, after the shooting is finished, it is possible to edit the video and adjust the position and time, but when there is a connection between the movements of two or more people, such as the timing of movements during a dance or the height of hands when shaking hands, it is difficult to adjust the discrepancy by simply changing the time or position after the fact.In contrast, the shot video selection device 1A according to the modified example can provide a return video to the performer in the studio, so that the performer can effectively adjust the timing of movements and the height of hands during studio shooting, and the 3D model can be completed efficiently.

[0063] Second embodiment Next, a captured video selection device according to a second embodiment will be described with reference to Fig. 8. Below, descriptions of configurations similar to those in Fig. 1 will be omitted as appropriate. The captured video selection device 1B according to the second embodiment includes a camera selection unit 41B and a 3D model obtainable area determination unit 42.

[0064] The 3D model obtainable area determination unit 42 determines whether the subject is included in the 3D model obtainable area of ​​the studio, that is, whether the subject is not protruding from the 3D model obtainable area of ​​the studio. The 3D model obtainable area is an area where a desired 3D model of the subject can be obtained without any loss. Here, the area within the shooting range of each camera that can be used as a return image in the studio is called the shootable area for each camera. The area within the shooting range of all cameras that can be used as a return image in the studio (the AND area of ​​each shootable area) is considered to be the 3D model obtainable area.

[0065] The 3D model obtainable area determination unit 42 receives camera image coordinates (two-dimensional coordinates) as posture information of the target subject from an external device (not shown). That is, the 3D model obtainable area determination unit 42 obtains three-dimensional position information P j For each camera C i The camera image coordinates I of each part j of the subject transformed into the image coordinate system (i=1,2,…,N) ij Get the.

[0066] The 3D model obtainable area determination unit 42 determines the camera image coordinates I ij Using the pixel information of each camera, it is determined for each camera whether each part j of the subject is within the photographable area of ​​the camera. Information of a camera that is determined to have at least one part j of the target subject Ha within the photographable area is output to the camera selection unit 41B.

[0067] Here, the pixel information of the camera indicates information on the number of pixels in the horizontal and vertical directions of the camera. As an example, the number of pixels in the horizontal direction of the camera is set to W, and the number of pixels in the vertical direction is set to H. Regarding the coordinates (horizontal direction: u, vertical direction: v) of a given part of the subject in the image coordinate system of each camera, if the horizontal coordinate value of the part is within 0 to W and the vertical coordinate value is within 0 to H, then the part is within the imageable area, and if not, the part is outside the imageable area.

[0068] The 3D model obtainable area determination unit 42 performs an inside / outside area determination process for indicating whether each part of the subject is within or outside the imageable area for the coordinates (horizontal direction: u, vertical direction: v) of the part set on the subject in the specified camera. In addition, the 3D model obtainable area determination unit 42 obtains information on parts within the imageable area and information on parts outside the imageable area in this inside / outside area determination process.

[0069] Furthermore, the 3D model obtainable area determination unit 42 obtains information on parts within the image-captureable area and information on parts outside the image-captureable area for each camera. When all of the subject's body parts can be photographed by all of the cameras, the 3D model obtainable area determination unit 42 determines that each subject body part j is within the 3D model obtainable area of ​​the studio in which the target subject is located.

[0070] The camera selection unit 41B receives, from an external device not shown, posture information (three-dimensional coordinates) of the target subject and posture information (three-dimensional coordinates) of a subject recorded separately, as well as the determination result from the 3D model obtainable area determination unit 42. The camera selection unit 41B selects a camera C that is determined by the 3D model obtainable area determination unit 42 to have at least one or more parts j of the target subject Ha within the photographable area. The camera selection unit 41B selects a camera C that is determined to have at least one or more parts j of the target subject Ha within the photographable area, based on the priority R iIn this way, by including as a condition that at least one part of the subject is within the captureable area of ​​the camera, it is possible to reliably prevent the selection of an image from a camera that does not show the subject at all.

[0071] (Modification 1 of the captured video selection device according to the second embodiment) A captured image selection device 1B according to the modified example can include an image processing unit 6 shown in Fig. 6. This captured image selection device 1B according to the modified example 1 includes a condition that the camera must have at least one location of the subject within the captureable area, thereby making it possible to prevent an image from a camera that does not show the subject at all from being used as the returned image.

[0072] (Modification 2 of the shot video selection device according to the second embodiment) As shown in FIG. 9, the captured video selection device 1C according to the modified example includes a camera selection unit 41B, a 3D model obtainable area determination unit 42C, and an image processing unit 6C. In the captured video selection device 1C according to the modified example 2, when it is not determined that all of the subject's body parts (j=1 to M) are within the 3D model obtainable area, the 3D model obtainable area determination unit 42C acquires information on the subject body parts that are outside the 3D model obtainable area. In this case, the 3D model obtainable area determination unit 42C notifies the camera selection unit 41B and the image synthesis unit 62C of the information on the subject body parts that are outside the 3D model obtainable area. Here, the information on the subject body parts that are outside the 3D model obtainable area includes not only the identification information of the body parts through the posture information of the subject, but also the identification information of the camera that photographed the body parts. When camera selection unit 41B acquires information about a part of the subject that is outside the 3D model obtainable area from 3D model obtainable area determination unit 42C, it selects the camera that captured the part outside the 3D model obtainable area based on the information about the part of the subject that was determined to be outside the 3D model obtainable area.

[0073] The image synthesizer 62C of the image processor 6C differs from the image synthesizer 62 of the image processor 6 shown in FIG. 6 in the following respects. First, the image synthesis unit 62C differs in that it acquires information on subject parts that are outside the 3D model obtainable area from the 3D model obtainable area determination unit 42C. Second, the image synthesis unit 62C differs in that it synthesizes a separately shot image of the subject with the image of the target subject captured by the camera selected by the camera selection unit 41B, and then attaches a marker (mark) to the part outside the image obtainable area. That is, the image synthesis unit 62C synthesizes the separately shot image of the subject with the in-studio camera image using the extraction result by the subject extraction unit 61, and creates an image to which the result of the 3D model obtainable area determination unit 42C is added.

[0074] The flow of image synthesis by this image synthesis unit 62C is the same as in Fig. 7, and images I1 (see Fig. 7), I2, and I3 are created in sequence. After that, the image synthesis unit 62C first extracts a part that is outside the area determined by the 3D model obtainable area determination unit 42C for the camera used in image I3. Next, the image synthesis unit 62C adds an image of a circle or a point centered on the image coordinates of the part outside the area to image I3 by using the posture information (camera image coordinates) for that part.

[0075] 10 is a diagram showing an example of an image I4 to which circles and dots are added. A dot 320 indicating the joint point of the right wrist, a circle 321 with the dot as its center, a dot 340 indicating the joint point of the right ankle 340, and a circle 341 with the dot as its center are added to the image I4. The image I4 shows that the right wrist and right ankle of the subject Ha are protruding outside the area where a 3D model can be acquired. Note that the subject shot separately is not shown. According to the shot video selection device 1C according to the second modification, it is possible to call the attention of the subject Ha (performer) in the studio that there is a part of the subject that is protruding from the area where a 3D model can be acquired.

[0076] Third embodiment The captured video selection devices 1, 1A, 1B, and 1C according to the above-described embodiments and modifications can be configured as a captured video display device that displays the returned video within the studio in combination with various studio equipment (cameras, recorders, monitors, etc.). Hereinafter, as a third embodiment, a captured image display device incorporating a captured image selection device 1C according to Modification 2 of the second embodiment will be described with reference to Fig. 11 (and Fig. 9, etc. as appropriate). Note that descriptions of the same configuration as in Fig. 9 will be omitted as appropriate. As shown in FIG. 11, the captured image display device 2 includes a photographing device 10, a recording device 20, a posture information acquisition device 30, a camera selection device 40 (corresponding to the captured image selection device 1C), a video receiving device 50, and a display device 60.

[0077] The photographing device 10 has a plurality of cameras including a main camera 11a for posture estimation and a sub-camera 11b for posture estimation. The cameras C that can be used to send back images from within the studio are all assumed to have the same number of pixels. For example, the number of pixels in the horizontal direction of each camera is W, and the number of pixels in the vertical direction is H. The posture estimation main camera 11a and the posture estimation sub-camera 11b are dedicated cameras for estimating the posture of the subject Ha in the volumetric capture studio. In the captured image display device 2, the position where the posture estimation main camera 11a is installed is processed as the front. In the example shown in FIG. 2, the angle between the posture estimation main camera 11a and the posture estimation sub-camera 11b is 90 degrees with respect to the center of the studio, but the angle between the posture estimation main camera 11a and the posture estimation sub-camera 11b can be changed within a range from -90 degrees to 90 degrees. If the image size of the posture estimation camera is the same as that of the camera C, the posture estimation camera can also be used as a camera to send back images in the studio.

[0078] The recording device 20 records a previously recorded image of the subject Hb and three-dimensional position information P jThe recording device 20 records the images captured in advance by all the cameras and the posture information of the subject in the images. The posture information acquisition device 30 acquires three-dimensional position information P for each part j of the subject as posture information of the subject from the image signals of the posture estimation main camera 11a and the posture estimation sub-camera 11b. j and projecting it onto the image coordinate system of each camera C.

[0079] As shown in FIG. 11, posture information acquisition device 30 includes a recorded image posture information acquisition section 31, a posture estimation camera image receiving section 32, a posture information acquisition section 33, and a posture information conversion section . The recorded image posture information acquisition unit 31 acquires posture information of a previously recorded image from the recording device 20. The recorded image posture information acquisition unit 31 acquires three-dimensional position information P j The recorded image posture information acquisition unit 31 acquires the three-dimensional position information P j to the camera selection unit 41B.

[0080] The posture estimation camera image receiving unit 32 receives images from two cameras used for posture estimation. The posture estimation camera image receiving unit 32 receives image signals from the posture estimation main camera 11a and the posture estimation sub-camera 11b, and outputs them to the posture information acquiring unit 3. The posture information acquisition unit 33 acquires three-dimensional position information P for each part j of the target object Ha using a posture estimation algorithm from the image signals of the posture estimation main camera 11a and the posture estimation sub-camera 11b. j is acquired as studio video posture information. Note that the posture estimation algorithm has been explained in the description of the shot video selection device 1 according to the first embodiment, so further explanation will be omitted. The posture information acquisition unit 33 acquires three-dimensional position information P j to the camera selection unit 41B and the orientation information conversion unit 34. The orientation information acquisition unit 33 also outputs the three-dimensional position information P jis output to the video receiving device 50.

[0081] The orientation information conversion unit 34 projects the studio image orientation information acquired from the orientation information acquisition unit 33 onto the image coordinate system of each camera. The orientation information (two-dimensional coordinates) projected onto the image coordinate system of each camera is output to the 3D model acquireable area determination unit 42C. This orientation information conversion unit 34 converts and projects the three-dimensional coordinate points of each part of the subject acquired by the orientation information acquisition unit 33 into points of camera image coordinates by using the position information and parameters of each camera in the studio.

[0082] Here, the conversion method will be described. As a premise, the number of cameras available for sending back images in the studio is set to N, and the number of parts of the subject acquired by the orientation information acquisition unit 33 is set to M. First, the orientation information conversion unit 34 converts the horizontal tilt θ of each camera based on the orientation of the camera set as the main camera out of the two cameras used for orientation estimation. ci , and the vertical tilt φ i and position C in the world coordinate system i (C ix ,C iy ,C iz ), where i is the camera number (1, 2, ..., N). The posture information conversion unit 34 then obtains the three-dimensional coordinate points P j (P jx ,P jy ,P jz ) is expressed as the camera image coordinate point A by the following equation (5). ij (A ijx ,A ijy ,A ijz ), where j is each subject part (1, 2, ..., M).

[0083]

number

[0084] Here, the roll angle of the camera (Z-axis rotation) is assumed to be 0 degrees. Next, the posture information conversion unit 34 converts the position of the point A in the camera image coordinate system into the position of the ij (A ijx ,A ijy ,A ijz ) to point N in normalized image coordinates ij (N ijx ,N ijy ) to

[0085] N ijx =A ijx / A ijz … Formula (6) N ijy =A ijy / A ijz … Formula (7)

[0086] Finally, the posture information conversion unit 34 converts the point N ij (N ijx ,N ijy ) is expressed as the image coordinate point I ij (u ij ,v jy ) to

[0087]

number

[0088] where f is the focal length of the camera and δ u is the physical spacing of pixels in the horizontal direction, δ v is the physical spacing of pixels in the vertical direction, (c u ,c v ) is the position of the intersection of the optical axis and the image plane in the image coordinate system (image center). These are internal parameters of the camera and can be obtained by camera calibration. By performing this transformation for all cameras and all parts of the subject, the parts of the subject can be projected onto the image coordinates of each camera.

[0089] The camera selection device 40 includes a camera selection section 41B in the captured image selection device 1C described above, and a 3D model obtainable area determination section 42C. The camera selection unit 41B selects the camera to be displayed as a return image based on the posture information of the subject shot separately acquired by the recorded video posture information acquisition unit 31 and the posture information of the subject in the studio acquired by the posture information acquisition unit 33. The 3D model obtainable area determination unit 42C is the same as the 3D model obtainable area determination unit 42C described in the captured image selection device 1C related to variant example 2 of the second embodiment, except that the 3D model obtainable area determination unit 42C obtains camera image coordinates (two-dimensional coordinates) as posture information of the target subject from the posture information conversion unit 34, and therefore further explanation will be omitted.

[0090] The video receiving device 50 receives and records a plurality of videos of a subject Ha captured by a plurality of cameras C. As shown in FIG. 11, the video receiving device 50 includes an all-camera video receiving unit 51 and a recording unit 52. The all-camera video receiving unit 51 receives camera video from all cameras in the studio that can be used as return video. If it is not necessary to record video from all cameras, the all-camera video receiving unit 51 can be substituted by, for example, a switcher. In that case, only the video from the camera selected by the camera selection unit 41 is output from the switcher.

[0091] The recording unit 52 records the camera images received by the all-camera image receiving unit 51. At the same time, by recording the posture information of the subject acquired by the posture information acquiring unit 33, the subject currently being filmed in the studio can be displayed as a separately shot subject image when another performer shoots later. If the cameras have built-in memories or connected recording devices, all camera images may be recorded for each camera rather than all at once by the recording unit 52.

[0092] The display device 60 combines the image of the target subject Ha captured by the camera C selected by the camera selection unit 41B with a separately recorded image of the subject Hb, and then marks 321, 341 on the body part j outside the imageable area and displays it on the display unit 63.

[0093] Display device 60 includes a subject extraction unit 61, an image synthesis unit 62C, and a display unit 63. Of these, subject extraction unit 61 and image synthesis unit 62C are similar components to image processing unit 6C of captured image selection device 1C according to Modification 2 of the second embodiment. That is, the subject extraction unit 61 extracts a subject from a separately recorded video of subject Hb that corresponds to the camera selected by the camera selection unit 41B, and also extracts a subject from a camera video that corresponds to the camera selected by the camera selection unit 41B. In addition, the image synthesis unit 62C synthesizes and creates an image to be displayed as the returned image using the subject extraction results for the separately recorded image of subject Hb and the image of the target subject Ha, and information from the camera that has been determined by the 3D model obtainable area determination unit 42C to contain a body part outside the photographable area. Here, the flow of image synthesis by image synthesis unit 62C is the same as in Fig. 7, in which image I1, image I2, and image I3 are created in sequence, and image I4 is created by adding circles and dots to image I3 as in Fig. 10. Image synthesis unit 62C switches images by updating the image currently displayed on display unit 63 to image I4.

[0094] The display unit 63 displays the created composite image. The display unit 63 displays the image updated by the video composition unit 62C. As the display unit 63, for example, a general display type monitor device such as a liquid crystal display can be used. Although one monitor device may be used, in this embodiment, a plurality of monitor devices are installed so that the images can be viewed from various directions in the studio. Note that, in addition to the display type monitor device, a projector may also be used as the monitor device.

[0095] The display device 60 switches the camera to be used if the camera selected by the camera selection unit 41B is different from the camera currently being displayed as the returned image. If the subject in the studio is not overlapped by a separately shot subject, and there is no camera that has at least one part of the subject within the photographable area, the display device 60 will not switch the camera. Here, in order to prevent frequent camera switching, a restriction can be set such that the camera is not switched for several seconds (for example, 2 seconds) after the camera is switched.

[0096] [Operation of the captured image display device] Next, the operation of the captured image display device 2 will be described with reference to Fig. 12 (and Fig. 11 as necessary). Fig. 12 is a flow chart showing the flow of processing by the captured image display device of Fig. 11. In step S1, the posture estimation camera image receiving unit 32 of the posture information acquisition device 30 acquires images from two posture estimation cameras (posture estimation main camera 11a and posture estimation sub-camera 11b) installed in the studio. In step S2, all-camera video receiving section 51 of video receiving device 50 acquires videos from cameras other than the posture estimation camera installed in the studio. In step S3, the posture information acquisition device 30 and the display device 60 are connected to the recording device 20 in order to acquire posture information and camera images of the subject that have been recorded separately in advance. The processing order of steps S1 to S3 is arbitrary, and steps S1 to S3 can also be performed in parallel.

[0097] Following step S3, recorded video posture information acquisition section 31 of posture information acquisition device 30 acquires pre-recorded posture information of the subject stored in recording device 20 (step S4). Here, the posture information includes the three-dimensional coordinates of each part of the subject for the number of recorded frames. Following step S3, subject extraction unit 61 of display device 60 acquires pre-recorded video stored in recording device 20 (step S5). The video acquired by subject extraction unit 61 is assumed to be acquired in sequence starting from the first frame, with the playback start timing being synchronized for all cameras. Note that the processing order of steps S4 to S5 is arbitrary.

[0098] Meanwhile, following step S1, posture information acquisition unit 33 of posture information acquisition device 30 acquires three-dimensional coordinate information of each part of the subject in the studio by utilizing a posture estimation algorithm from the image of the posture estimation camera received in step S1 (step S6). Then, posture information conversion unit 34 of posture information acquisition device 30 converts the three-dimensional coordinate information of each part of the subject acquired in step S6 from the world coordinate system into each camera image coordinate using the camera parameters of each camera (step S7).

[0099] Next, the 3D model obtainable area determination unit 42C of the camera selection device 40 performs an inside / outside area determination process indicating whether the subject is inside or outside the imageable area based on the coordinates of each part of the subject in each camera image coordinate acquired in step S7 (step S8). At the same time, the 3D model obtainable area determination unit 42C records whether each part is inside or outside the imageable area for each camera.

[0100] Then, camera selection unit 41B of camera selection device 40 selects a camera to be displayed in the studio based on the three-dimensional posture information of the studio subject obtained in step S6, the posture information of the subject from the pre-recording obtained in step S4, and the information on whether each part of the subject is inside or outside the photographable area of ​​each camera obtained in step S8 (step S9).

[0101] Following step S9, subject extraction unit 61 of display device 60 acquires an image corresponding to the camera selected in step S9 from the studio camera video received in step S2 (step S10). Also, following step S9, subject extraction unit 61 acquires an image corresponding to the camera selected in step S9 from the pre-recorded video acquired in step S5 (step S11). Note that the processing order of steps S10 to S11 is arbitrary. Following steps S10 and S11, subject extraction section 61 extracts the subject from the images acquired in steps S10 and S11, respectively, and separates the image into the subject and non-subject (step S12).

[0102] Following step S12, the image synthesis unit 62C of the display device 60 synthesizes the subject in the pre-recorded image with the in-studio image using the subject information extracted in step S12. Furthermore, the image synthesis unit 62C adds marks such as circles and dots to the parts that are outside the 3D model obtainable area and appear in the image based on the camera image coordinates of each part acquired in step S7 (step S13). Then, the display unit 63 of the display device 60 displays the image created in step S13 (step S14).

[0103] If the shooting is not completed (step S15: No), the process returns to step S1. On the other hand, for example, if the pre-recorded video has been played back up to the final frame (step S15: Yes), the shooting is terminated. Alternatively, for example, if the camera shooting in the studio has been completed (step S15: Yes), the shooting is terminated. Note that the operator of the shot video display device 2 may determine the end of the studio shooting and force the end by inputting, for example, "end" as a signal to end the program.

[0104] As described above, the captured image display device 2 selects the camera of the returned image based on the posture information of the subject, and plays and displays it. This captured image display device 2 adds the image of a performer who was recorded in advance in the same studio to the returned image, so that the performer in the studio can perform together with the performer who was recorded separately, with their movements synchronized. In addition, the captured image display device 2 provides a return image that combines a separately shot subject based on the image captured by the camera and the posture information of the subject shown in the image, making it possible to create a 3D model in which the movement timing of multiple people is synchronized, even if the subjects are shot one by one. Therefore, when creating a work that includes scenes in which multiple performers perform related actions such as shaking hands, high-fiving, and dancing, it becomes easier to synchronize the movement timing than before. Furthermore, the captured image display device 2 calls attention to the need to prevent the subject from going out of frame of the camera in the studio during shooting, thereby making it possible to prevent defects from occurring in the three-dimensional model to be created.

[0105] In this way, using the captured image display device 2, the performers (subjects) in the studio can check the camera images that are composited with pre-recorded images of the performers and perform while matching their movements, allowing the filming to proceed smoothly. On the other hand, in conventional volumetric studios, video monitors for actors to check their own performances were rarely installed, and it was not possible to satisfactorily check not only the separately shot images of the actors but also the images of the subject currently being shot in the studio. Therefore, in order for the studio actors to check the images being shot, they had to go outside the studio once and play back the recorded images in another room to check them. Even if a monitor was rarely installed in the studio, it was only one monitor for displaying images shot by a camera fixed at one viewpoint. Therefore, conventional studio equipment could not provide appropriate images for actors to check their own performances, and furthermore could not provide images that could be used as a reference for matching the performances of multiple actors. In contrast, the filmed image display device 2 has a display unit 63 installed in the studio, allowing the performers to check their own movements and those of the other performers in real time. Therefore, the filmed image display device 2 improves the time efficiency of studio filming and contributes to making it easier for the performers in the studio to perform.

[0106] Although the shot image selection device and the shot image display device according to the embodiment of the present invention have been described above, the gist of the present invention is not limited to these descriptions and should be broadly interpreted based on the claims. Furthermore, it goes without saying that various changes and modifications based on these descriptions are also included in the gist of the present invention.

[0107] For example, the captured image display device 2 has been described as sending back and adding images of performers pre-recorded in the same studio, but the image of a separately recorded subject may be taken in a separate volumetric capture studio with the same camera arrangement as the studio where the target subject is located. In this case, the shooting time required to combine the performances of multiple performers can be further reduced, and the subject in the studio where the target subject is located can be efficiently shot. In addition, when shooting three or more performers, the time can be further reduced by splitting the performers into two halves in the same studio set and recording them alternately.

[0108] Furthermore, as the third embodiment, a captured image display device incorporating a captured image selection device 1C relating to variant example 2 of the second embodiment has been described. However, this is not limited to this, and instead of variant example 2, it is also possible to incorporate a captured image selection device relating to variant example 1, or the first embodiment, or a variant example of the first embodiment, into the captured image display device.

[0109] In addition, in each of the above-mentioned embodiments, the captured image selection device has been described as an independent piece of hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program (captured image selection program) that operates hardware resources such as a CPU, memory, and hard disk of a computer as the above-mentioned device. This program may be distributed via a communication line, or may be written on a recording medium such as a CD-ROM or a flash memory and distributed. [Explanation of symbols]

[0110] 1, 1A, 1B, 1C Shooting image selection device 2. Recorded image display device 6,6C Image processing section 10. Imaging Device 11a Main camera for posture estimation (posture estimation camera) 11b Sub-camera for posture estimation (Posture estimation camera) 20 Recording Device 30 Attitude information acquisition device 31 Recorded video posture information acquisition unit 32 Attitude estimation camera image receiver 33 Posture information acquisition unit 34 Posture information conversion unit 40 Camera selection device 41,41B Camera selection section 42 3D model acquisition area determination unit 50 Video receiving device 51 All camera image receiving unit 52 Recording Section 60 Display device 61 Subject extraction section 62,62C Video synthesis section 63 Display section 100 Primary selection section 110 Object shape calculation unit 120 Camera relative position calculation unit 130 Exclusion Camera Judgment Unit 200 Second selection section 210 Subject angle calculation section 212 Straight line calculation section 214 Angle Calculation Unit 220 Difference index calculation section 230 Priority calculation unit C,C i camera

Claims

1. A video selection device that selects one of a plurality of video images captured by a plurality of cameras surrounding one target subject in a volumetric capture studio as a video image to be combined with a video image of at least one subject separately recorded in the volumetric capture studio, the device comprising: A captured image selection device comprising a camera selection unit that selects a camera that is positioned so that the separately recorded subject does not obstruct the target subject during image synthesis and that is positioned so that it can capture the front side of the target subject.

2. The camera selection unit a primary selection unit that uses posture information, which is three-dimensional position information for each body part of the target subject and the separately recorded subject estimated by a posture estimation camera, and position information of the multiple cameras, to exclude cameras whose circle representing the target subject as viewed from the camera is in the shadow of the circle representing the separately recorded subject, based on the positions of circles obtained by projecting the target subject and the separately recorded subject onto a plane corresponding to the floor of a studio and the positions of each camera on the plane, and selects the remaining cameras; a subject angle calculation unit that calculates a subject angle indicating a front direction of each subject with respect to a reference line that is predetermined on the plane using posture information of the target subject and the separately recorded subject; a difference index calculation unit that calculates a difference between a camera angle indicating a front direction of each camera with respect to the reference line and the subject angle, as an index indicating that a camera is in a position capable of photographing a front side of the subject, the difference between the camera angle and the subject angle for each subject; The video footage selection device of claim 1, further comprising a priority calculation unit that calculates a priority for each camera by weighting and adding a first index indicating the difference between the camera angle and the subject angle for the target subject and a second index indicating the difference between the camera angle and the subject angle for the separately recorded subject, and selects the camera with the smallest priority from among the cameras selected by the primary selection unit.

3. The primary selection unit, a subject shape calculation unit that calculates center coordinates and radii of circles representing the target subject on the plane and the separately recorded subject; a camera relative position calculation unit that calculates, for each camera, a gradient of a tangent line from the camera to each of the circles and a distance from the camera to a center of each of the circles, using position information of the plurality of cameras and a calculation result by the subject figure calculation unit; The video footage selection device of claim 2, further comprising a camera exclusion determination unit that excludes cameras for which the slopes of two tangents to a circle representing the target subject are determined to be greater than the slope of one tangent to the circle representing the separately recorded subject and less than the slope of the other tangent, and selects the remaining cameras.

4. The subject angle calculation unit A straight line calculation unit calculates, for each of the target subject and the separately recorded subject, a first straight line that is a diameter of a circle that represents the subject on the plane and indicates a width direction of the subject, and a second straight line that passes through the center of the circle and is perpendicular to the first straight line, using posture information of each subject; The image selection device of claim 2, further comprising an angle calculation unit that calculates the angle between the reference line and the second straight line for each subject, and calculates the subject angle based on the calculated angle and posture information of each subject.

5. a 3D model obtainable area determination unit which obtains camera image coordinates of each part of the subject by converting three-dimensional position information of the part of the subject into an image coordinate system of each camera, and determines for each camera whether each part of the subject is within a photographable area of ​​the camera using the camera image coordinates of each part of the subject and pixel information of each camera, obtains information on parts within the photographable area and information on parts outside the photographable area for each camera, and determines that each part of the subject is within a 3D model obtainable area of ​​the studio in which the target subject is located if all of the parts of the subject can be photographed by all of the cameras; The photographed image selection device according to claim 3, characterized in that the camera selection unit selects a camera for which the 3D model obtainable area determination unit has determined that at least one or more parts of the target subject are within the photographable area.

6. when it is not determined that all of the body parts of the subject are within the 3D model obtainable area, the 3D model obtainable area determination unit obtains information on body parts of the subject that are outside the 3D model obtainable area, The captured image selection device of claim 5, characterized in that the camera selection unit selects a camera that captured an area outside the area where a 3D model can be obtained, based on information about the subject area that is determined to be outside the area where a 3D model can be obtained.

7. The photographed image selection device according to claim 6 ; an imaging device having a plurality of cameras including the posture estimation camera; A recording device that records a pre-recorded image of the subject in the separate recording and three-dimensional position information of each part of the subject in the separate recording; a posture information acquisition device that acquires three-dimensional position information of each part of a subject as posture information of the subject from a video signal of the posture estimation camera, and projects the information onto an image coordinate system of each camera; a video receiving device that receives and records a plurality of images of the target subject captured by the plurality of cameras; a display device that combines a video of the target subject captured by a camera selected by the camera selection unit with a separately recorded video of the subject that was recorded in advance, and displays the video on a display unit with marks placed on areas outside the area where a 3D model can be obtained.

8. The posture information acquisition device includes: a recorded image posture information acquisition unit that acquires three-dimensional position information of each part of the subject recorded separately from the recording device as recorded image posture information; a posture estimation camera image receiving unit for receiving an image signal of the posture estimation camera; a posture information acquisition unit that acquires, as studio image posture information, three-dimensional position information for each part of the subject of the target from the video signal of the posture estimation camera by using a posture estimation algorithm; 8. The captured image display device according to claim 7, further comprising an orientation information conversion section for projecting the studio image orientation information onto an image coordinate system of each camera.

9. The display device includes: a subject extraction unit that extracts a subject from the separately recorded video of the subject that corresponds to the camera selected by the camera selection unit and extracts a subject from the camera video that corresponds to the camera selected by the camera selection unit; a video synthesis unit that synthesizes and creates an image to be displayed as a return video using the subject extraction results for the separately recorded subject video and the target subject video, and information from a camera that is determined by the 3D model obtainable area determination unit to have a part outside the 3D model obtainable area; and 9. The captured image display device according to claim 7, further comprising: the display unit for displaying the generated composite image.

10. A shot video selection program for causing a computer to function as the shot video selection device according to any one of claims 1 to 6.

Images