Image processing apparatus and image processing system
By obtaining the location information of the camera device, 2D videos of the virtual viewpoint are generated and synthesized with the videos of the real camera device, the problems of high cost and insufficient realism of 3D CG videos are solved, and low-cost and strong realism are achieved.
Patent Information
- Application Number
- CN202380029653.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-03-15
- Publication Date
- 2024-11-15
AI Technical Summary
In 2D transmission, when synthesizing the foreground and background of 3D CG video, it is expensive and takes a long time to generate 3D CG video. The 3D CG video lacks a sense of reality when used as a background, making it difficult to create a sense of liveness.
By acquiring the position information of the imaging device as the virtual viewpoint information, a 2D video viewed through the virtual image viewpoint is generated, and it is synthesized with the 2D video generated by the real image capture device to generate a synthetic image to achieve low-cost real 2D transmission.
It realizes 2D transmission that generates a sense of reality at low cost and low latency, improves the authenticity of background video, and has a high naturalness of the foreground and background superposition, so that users can feel the sense of scene.
Smart Images

Figure CN118974782A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image processing device and an image processing system, and more particularly, to an image processing device and an image processing system capable of realizing 2D transmission that can produce a sense of reality at low cost. Background Art
[0002] There is a technology for providing a video of a free viewpoint by generating a 3D model of a subject from a moving image imaged from multiple viewpoints and generating a virtual viewpoint video of the 3D model corresponding to any viewpoint (virtual viewpoint) (for example, see Patent Document 1). This technology is also called volumetric capture, etc. There are the following methods: a method in which a user at a transmission destination determines a virtual viewpoint of a 3D model of a subject and transmits a video so that each user can freely change the viewpoint (hereinafter, referred to as 3D transmission); and a method in which a transmission side determines a virtual viewpoint and transmits a video of the same virtual viewpoint to multiple users (hereinafter, referred to as 2D transmission).
[0003] As a method for synthesizing a foreground video and a background video, for example, there is a chroma key synthesis technique of synthesizing a foreground video and a background video obtained by imaging a person in a studio (for example, see Patent Document 2).
[0004] Reference List
[0005] Patent Literature
[0006] Patent Document 1: WO 2018 / 150933
[0007] Patent Document 2: Japanese Patent Application Publication No. 2012-175128 Summary of the invention
[0008] Problems to be solved by the present invention
[0009] In the above-mentioned 2D transmission, when trying to synthesize a foreground video of a 3D model captured by volume and a background video generated by a 3D CG video, it takes a long time to generate the 3D CG video, and a large cost is incurred. In addition, when a 3D CG video is used as a background, the person in the foreground is as real as a real image through volume capture, while the background video lacks realism. In addition, in the case where a background video based on a pre-created 3D CG video is synthesized, it looks like a pre-recorded video, and it is difficult to create a sense of presence.
[0010] The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to realize 2D transmission capable of producing a sense of reality at a low cost.
[0011] Solution to the problem
[0012] An image processing device according to a first aspect of the present disclosure includes: a 2D video generation unit that obtains camera position information of a camera that performs imaging at a first place as virtual viewpoint information, and generates a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second place different from the first place from the viewpoint of the camera.
[0013] An image processing system according to a second aspect of the present disclosure includes: a 2D video generating device that acquires camera device position information of a camera device that performs imaging at a first place as virtual viewpoint information, and generates a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second place different from the first place from the viewpoint of the camera device; and a video synthesizing device that generates a synthesized image obtained by synthesizing a 2D video generated by the 2D video generating device and a 2D video generated by the camera device.
[0014] In the first and second aspects of the present disclosure, camera position information of a camera that performs imaging at a first location is acquired as virtual viewpoint information, and a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second location different from the first location from the viewpoint of the camera is generated. In addition, in the second aspect of the present disclosure, a composite image is generated by synthesizing a 2D video generated by a 2D video generating device and a 2D video generated by the camera.
[0015] Note that each of the first image processing system and the image processing system according to the second aspect of the present disclosure can be realized by causing a computer to execute a program. The program executed by the computer can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.
[0016] The image processing apparatus and the image processing system may be independent apparatuses, or may be internal blocks comprising one device. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a diagram for describing an overview of volume capture.
[0018] Figure 2 is a diagram for describing an overview of volume capture.
[0019] Figure 3 It is a block diagram for describing a first embodiment of an image processing system to which the present technology is applied.
[0020] Figure 4 Is used to describe Figure 3 Figure 4. Illustration of the image processing system in the transmission of composite video.
[0021] Figure 5 Is used to describe Figure 3 Figure 4. Illustration of the image processing system in the transmission of composite video.
[0022] Figure 6 Is used to describe Figure 3 Figure 4. Illustration of the image processing system in the transmission of composite video.
[0023] Figure 7 Is used to describe Figure 3 Figure 4. Illustration of the image processing system in the transmission of composite video.
[0024] Figure 8 Is used to describe Figure 3 Figure 4. Illustration of the image processing system in the transmission of composite video.
[0025] Fig. 9 is a flow chart for describing the volumetric video generation process.
[0026] Fig.10 is a flow chart describing the volumetric 2D video generation process.
[0027] Fig.11 is a flowchart for describing the composite video generation process.
[0028] Fig.12 Is used to describe Fig.11 Flowchart of details of the video synthesis processing in step S54.
[0029] Fig.13 Yes Figure 3 A diagram comparing an image processing system with another system.
[0030] Fig.14 is a block diagram showing a modification of the first embodiment of the image processing system.
[0031] Fig.15 is a block diagram showing a first configuration example of a second embodiment of an image processing system to which the present technology is applied.
[0032] Fig.16 : is a diagram for describing the operation of the first configuration example of the second embodiment.
[0033] Fig.17 is a block diagram illustrating a second configuration example of the second embodiment of the image processing system to which the present technology is applied.
[0034] Fig.18 : is a diagram showing an example of transmitting a video according to a second configuration example of the second embodiment.
[0035] Fig.19is a block diagram illustrating a third configuration example of the second embodiment of the image processing system to which the present technology is applied.
[0036] Fig. 20 : is a diagram showing an example of a composite video according to the third configuration example of the second embodiment.
[0037] Fig.21 This is a block diagram showing a third embodiment of an image processing system to which the present technology is applied.
[0038] Fig. 22 is a diagram for describing the origin operation in each coordinate setting mode.
[0039] Fig.23 is a diagram for describing origin processing in each coordinate setting mode.
[0040] Fig.24 is a diagram for describing a control example of the camera position of virtual viewpoint information in each coordinate setting mode.
[0041] Fig.25 is a diagram showing an example of a mode selection button and an origin position designation button.
[0042] Fig.26 1 is a flowchart for describing the virtual viewpoint information generation process performed by the background video device of the third embodiment.
[0043] Fig. 27 is a diagram showing a configuration example using a smartphone or a drone as a real camera device.
[0044] Fig.28 This is a block diagram showing a fourth embodiment of an image processing system to which the present technology is applied.
[0045] Fig.29 A diagram for describing illumination information acquired by a lighting sensor and lighting control information for controlling a lighting device.
[0046] Fig.30 : is a flowchart for describing the lighting control process performed by the image processing system 1 according to the fourth embodiment.
[0047] Fig.31 This is a block diagram showing a fifth embodiment of an image processing system to which the present technology is applied.
[0048] Fig.32 : is a figure which shows the installation example of the omnidirectional camera device and the wall display.
[0049] Fig.33 : is a flowchart for describing the omnidirectional video output process performed by the omnidirectional video output device according to the fifth embodiment.
[0050] Fig.34 Is used to describe Fig.33 Flowchart of the re-projection process of the omnidirectional video in step S152.
[0051] Fig.35 is a block diagram illustrating a configuration example of an embodiment of a computer to which the technology of the present disclosure is applied. DETAILED DESCRIPTION
[0052] Hereinafter, a mode (hereinafter referred to as an implementation mode) for executing the technology of the present disclosure will be described with reference to the accompanying drawings. Note that in this specification and the accompanying drawings, components having substantially the same functional configuration are represented by the same reference numerals, and redundant descriptions are omitted. The description is given in the following order.
[0053] 1. Overview of Volume Capture
[0054] 2. First Embodiment of Image Processing System
[0055] 3. Transmission of synthetic video by image processing system
[0056] 4. Volumetric Video Generation Process
[0057] 5. Volumetric 2D video generation process
[0058] 6. Synthetic video generation process
[0059] 7. Comparison with other systems
[0060] 8. Modification of the First Embodiment
[0061] 9. Second Embodiment of Image Processing System
[0062] 10. Third Embodiment of Image Processing System
[0063] 11. Fourth Embodiment of Image Processing System
[0064] 12. Fifth Embodiment of Image Processing System
[0065] 13. Summary of the Image Processing System of the Present Disclosure
[0066] 14. Computer Configuration Example
[0067] <1. Overview of Volume Capture>
[0068] The image processing system of the present disclosure relates to volumetric capture of a video (free viewpoint video) providing a free viewpoint by generating a 3D model of a subject from moving images imaged from multiple viewpoints and generating a virtual viewpoint video of the 3D model corresponding to any viewing position.
[0069] Therefore, first of all, we will refer to Figure 1 The generation of a 3D model of a subject and the display of a free viewpoint video using the 3D model are briefly described.
[0070] For example, a plurality of imaging images can be obtained by imaging a predetermined imaging space from the periphery of the predetermined imaging space by a plurality of imaging devices, in which a subject such as a person is set. The imaging images include, for example, moving images. Figure 1 In the example of , three imaging devices CAM1 to CAM3 are arranged to surround the subject #Ob1, but the number of imaging devices CAM is not limited to three, but is an arbitrary number. Since the number of imaging devices CAM at the time of imaging is the known number of viewpoints when generating a free viewpoint video, as the number increases, the free viewpoint video can be represented with higher accuracy. The subject #Ob1 is a person performing a predetermined movement.
[0071] A 3D object MO1 is generated by using imaging images obtained from a plurality of imaging devices CAM in different directions, and the 3D object MO1 is a 3D model (3D modeling) of a subject #Ob1 to be displayed in an imaging space. The 3D object MO1 can be generated by using, for example, a method such as a visual shell in which the three-dimensional shape of the subject is cut out by using imaging images in different directions.
[0072] Then, the 2D video displayed on the viewing device of the viewer is displayed by rendering the 3D object using data of one or more 3D objects among the one or more 3D objects existing in the imaging space (hereinafter, also referred to as 3D model data). Figure 1 An example is shown in which the viewing device is a display D1 or a head mounted display (HMD) D2.
[0073] Figure 2 An example of the data format of general 3D model data is shown.
[0074] The 3D model data is generally expressed by 3D shape data indicating the 3D shape (geometric information) of the object and texture data indicating color information of the object.
[0075] 3D shape data may be expressed, for example, in a point cloud format in which the three-dimensional position of a subject is represented by a collection of points, a 3D mesh format in which the three-dimensional position of a subject is represented by vertices and connections between vertices called a polygonal mesh, or a voxel format in which the three-dimensional position of a subject is represented by a collection of cubes called voxels.
[0076] Examples of texture data include a multi-texture format retained in an imaged image (two-dimensional texture image) imaged by each imaging device CAM, and a UV mapping format retained by expressing a two-dimensional texture image attached to each point or each polygon mesh as 3D shape data in a UV coordinate system.
[0077] like Figure 2 As shown in the upper part of the figure, the format of describing the 3D model data in a multi-texture format and 3D shape data retained in a plurality of imaging images P1 to P8 imaged by the imaging device CAM is a view-dependent format in which color information can be changed according to the virtual viewpoint (the position of the virtual camera device).
[0078] On the other hand, Figure 2 As shown in the lower part of , the format of describing the 3D model data by 3D shape data and UV mapping format (in which the texture information of the subject is mapped to the UV coordinate system) is a view-independent format, in which the color information is the same depending on the virtual viewpoint (the position of the virtual camera device).
[0079] <2. First Embodiment of Image Processing System>
[0080] Figure 3 This is a block diagram showing a first embodiment of an image processing system to which the present technology is applied.
[0081] Figure 3 The image processing system 1 is a video transmission system that transmits a synthesized video obtained by synthesizing a video of a subject (eg, a person) imaged in a volume studio and a background video imaged in a background imaging studio.
[0082] The background imaging system 11 and the monitor 12 of the image processing system 1 are installed in the background imaging studio. In addition, the volume imaging system 21, the volume 2D video generating device 22 and the monitor 23 of the image processing system 1 are installed in the volume studio. In addition, the image processing system 1 includes a video synthesis device 31 and a 2D video transmission device 32, the video synthesis device 31 is installed in the video synthesis center, and the 2D video transmission device 32 is installed in the transmission center.
[0083] The image processing system 1 transmits a composite video obtained by synthesizing a video imaged by a real camera (actual camera) as a background video and a 2D video of a person generated by using a volume capture technology in a volume studio to a client device 33 of a user. Both the video of the person imaged in the volume studio and the background video imaged in the background imaging studio are generated as moving images in real time (immediately) and at the same time. The composite video is also generated as a moving image in real time and transmitted to the client device 33 as a transmission video. Note that the transmission to the client device 33 can be transmitted in response to a request from the user (on demand).
[0084] For example, the background imaging studio, the volume studio, the video synthesis center, and the transmission center may be arranged close to each other in the same building, or may be arranged far away from each other. For example, data may be sent and received via a predetermined network such as a local area network, the Internet, a public telephone line network, a mobile communication network for wireless mobile bodies such as a so-called 4G line or 5G line, a digital satellite broadcast network, or a television broadcast network.
[0085] For simplicity, the location of the background imaging system 11 is an indoor background imaging studio, but is not limited to indoors and may be, for example, an outdoor imaging environment referred to as a location site. In addition, it is assumed that the video imaged in the background imaging studio images the background video of the person in the volumetric studio, but a portion of the video may be the foreground of the person in the volumetric studio.
[0086] The background imaging system 11 of the background imaging studio includes a camera 51R, a camera 51D, a camera motion detection sensor 52 , and a background video generating device 53 .
[0087] Camera 51R is an imaging device that images a color (RGB) 2D video as a background video. Camera 51D is an imaging device that detects the depth value (distance information) of the subject imaged by camera 51R and generates a depth video in which the detected depth value is stored as a pixel value. Camera 51D is adjusted and installed so that the optical axis coincides with camera 51R. Camera 51R and camera 51D may be a combined imaging device.
[0088] In the following description, for easy distinction, the camera 51D will be described as the RGB camera 51R and the camera 51D as the depth camera 51D. In addition, when the RGB camera 51R and the depth camera 51D are integrally represented, these cameras will be described as the real camera 51.
[0089] The camera motion detection sensor 52 is a sensor that obtains the camera position information of the real camera 51. The camera position information includes the position of the real camera 51 represented by the three-dimensional coordinates of (x, y, z) with a predetermined origin as a reference, the orientation of the real camera 51 represented by (pan, tilt, roll), and the zoom value represented by a value from 0 to 100%. "Pan" represents the orientation in the left and right direction, "tilt" represents the orientation in the up and down direction, and "roll" represents the rotation around the optical axis. For example, the camera motion detection sensor 52 is attached to a movable unit such as a camera platform. A sensor that obtains the camera position, camera orientation, and zoom value may be provided separately as the camera motion detection sensor 52.
[0090] The background video generating device 53 performs adjustment so that the 2D video supplied from the RGB camera device 51R and the depth video supplied from the depth camera device 51D have the same viewing angle. Then, after adjusting the viewing angle, the background video generating device 53 assigns a background imaging system ID and a frame number (FrameNo) to the 2D video and the depth video, and supplies the 2D video and the depth video to the video synthesizing device 31. Hereinafter, in order to facilitate distinction from other videos, the 2D video and the depth video supplied from the background video generating device 53 to the video synthesizing device 31 are respectively referred to as background video (RGB) and background video (Depth), and the set of background video (RGB) and background video (Depth) is referred to as background video (RGB-D).
[0091] In addition, the background video generating device 53 assigns the background imaging system ID and the frame number to the position, orientation and zoom value of the real camera 51 supplied from the camera motion detection sensor 52, and supplies the position, orientation and zoom value as virtual viewpoint information to the volumetric 2D video generating device 22. For the transmission of the virtual viewpoint information, any protocol can be used, and for example, the FreeD protocol for creating AR / VR content can be used.
[0092] The monitor 12 displays the composite video (RGB) supplied from the video composite device 31 in the video composite center. The composite video (RGB) displayed on the monitor 12 is a color (RGB) 2D video, and is the same video as the composite video (RGB) sent from the video composite device 31 to the 2D video transmission device 32. The composite video (RGB) displayed on the monitor 12 is a video to be confirmed by a person who is a performer in the background imaging studio.
[0093] The volume imaging system 21 of the volume studio includes N (N>1) camera devices 71 - 1 to 71 -N and a volume video generating device 72 .
[0094] The cameras 71-1 to 71-N are arranged around the imaging area to surround the person in the volume studio. Each of the cameras 71-1 to 71-N images the person as a subject, and supplies the imaging image obtained as a result to the volume video generation device 72. The camera parameters (external parameters and internal parameters) of the installation site including the cameras 71-1 to 71-N are known and supplied to the volume video generation device 72.
[0095] The volumetric video generating device 72 generates a 3D model of a person in the volumetric studio from the imaging images supplied from the cameras 71-1 to 71-N by using the volumetric capture technology. The volumetric video generating device 72 supplies the generated 3D model data of the person to the volumetric 2D video generating device 22. As described above, the 3D model data of the person includes 3D shape data and texture data.
[0096] The volume 2D video generating device 22 acquires virtual viewpoint information from the background video generating device 53, and acquires 3D model data of people in the volume studio from the volume video generating device 72. The virtual viewpoint information includes a background imaging system ID and a frame number.
[0097] The volumetric 2D video generating device 22 generates a 2D video and a depth video obtained by viewing a 3D model of a person in the volumetric studio from a virtual camera device based on the virtual viewpoint information, assigns a background imaging system ID and a frame number that are the same as the virtual viewpoint information, and supplies the 2D video and the depth video to the video synthesizing device 31. In order to distinguish from the above-mentioned background video (RGB) and background video (Depth), the 2D video and the depth video supplied from the volumetric 2D video generating device 22 to the video synthesizing device 31 are respectively described as volumetric 2D video (RGB) and volumetric 2D video (Depth), and the set of volumetric 2D video (RGB) and volumetric 2D video (Depth) is described as volumetric 2D video (RGB-D).
[0098] The monitor 23 displays the composite video (RGB) supplied from the video composite device 31 in the video composite center. The composite video (RGB) displayed on the monitor 23 is a color (RGB) 2D video, and is the same video as the composite video (RGB) sent from the video composite device 31 to the 2D video transmission device 32. The composite video (RGB) displayed on the monitor 23 is a video to be confirmed by a person who is a performer in the volume studio.
[0099] The video synthesizing device 31 generates a synthesized video (RGB) by synthesizing the background video (RGB-D) from the background video generating device 53 and the volumetric 2D video (RGB-D) from the volumetric 2D video generating device 22 having the same background imaging system ID and frame number. Specifically, the video synthesizing device 31 compares the depth value of the background video (RGB-D) at the same pixel position with the depth value of the volumetric 2D video (RGB-D), and generates a synthesized video (RGB) to give priority to closer subjects. The synthesized video (RGB) is a color (RGB) 2D video. The video synthesizing device 31 supplies the generated synthesized video (RGB) to the 2D video transmitting device 32, and also supplies the synthesized video to the monitor 12 of the background imaging studio and the monitor 23 of the volumetric studio.
[0100] The 2D video transmission device 32 sends (transmits) the synthesized video (RGB) sequentially supplied from the video synthesis device 31 as a transmission video to one or more client devices 33 via a predetermined network. Transmission from the 2D video transmission device 32 to each client device 33 can be performed, for example, via a predetermined network such as the Internet, a mobile communication network for wireless mobile bodies such as a so-called 4G line or 5G line, a digital satellite broadcast network, or a television broadcast network.
[0101] The client device 33 includes, for example, a personal computer, a smart phone, etc., acquires the synthesized video (RGB) from the 2D video transmission device 32 via a predetermined network, and displays the synthesized video on a predetermined display device. For example, the 2D video transmission device 32 compresses the synthesized video (RGB) sequentially supplied from the video synthesis device 31 every time interval, and places the synthesized video in a transmission server so that the synthesized video can be accessed from the client device 33 via a content delivery network (CDN). The client device 33 acquires and reproduces the synthesized video (RGB) placed on the transmission server via the CDN.
[0102] Each of the background video generating device 53, the volumetric video generating device 72, the volumetric 2D video generating device 22, the video synthesizing device 31 and the 2D video transmitting device 32 may be constituted by, for example, a server device, a dedicated image processing apparatus or the like.
[0103] The image processing system 1 of the first embodiment has the above configuration.
[0104] <3. Transmission of composite video by image processing system>
[0105] Will refer to Figures 4 to 8 The transmission of the composite video by the image processing system 1 is described.
[0106] Figure 4The process flow from imaging in the background imaging studio and volumetric studio to delivery of the composite video to the client device 33 is shown.
[0107] First, a predetermined position is set as an origin position in each of the background imaging studio and the volume studio. Any method can be used as a method for setting the origin position. For example, a method for moving the real camera 51 to a place in the background imaging studio that is desired to be set as the origin and pressing an origin setting button to set the current position of the real camera 51 as the origin can be used. In the volume studio, the predetermined position is also set as the origin position by a similar method or a different method.
[0108] In the background imaging studio, the real camera 51 images the person ACT1 and the background as the subject. More specifically, the RGB camera 51R images the person ACT1 and the background, and outputs the 2D video obtained as a result to the background video generating device 53. The depth camera 51D detects the distance between the person ACT1 and the background, and outputs the depth video to the background video generating device 53. The camera motion detection sensor 52 acquires the position, orientation, and zoom value of the real camera 51, and outputs the acquired position, orientation, and zoom value to the background video generating device 53.
[0109] The background video generating device 53 performs adjustment so that the 2D video and the depth video have the same viewing angle. The 2D video and the depth video after adjusting the viewing angle are the background video (RGB) and the background video (Depth). The background imaging system ID and the frame number are assigned to the background image (RGB) and the background image (Depth), and are output from the background video generating device 53 to the video synthesizing device 31.
[0110] In addition, the background video generating device 53 assigns the background imaging system ID and frame number to the position, orientation and zoom value of the real camera 51 supplied from the camera motion detection sensor 52, and outputs the position, orientation and zoom value as virtual viewpoint information to the volumetric 2D video generating device 22.
[0111] Figure 5 An output example of virtual viewpoint information is shown.
[0112] The origin position is set to position (x, y, z) = (X0, Y0, Z0) and orientation (pan, tilt, roll) = (pan0, TILT0, ROLL0). After setting the origin, when the camera motion detection sensor 52 detects the position and orientation of the real camera 51 as position (Xc, Yc, Zc) and orientation (panc, TILTc, ROLLc), the background video generation device 53 calculates the virtual viewpoint information as follows and outputs the virtual viewpoint information to the volume 2D video generation device 22.
[0113] Position (x, y, z) = (Xc-X0, Yc-Y0, Zc-Z0)
[0114] Orientation (pan, tilt, roll) = (panc-pan0, TILTc-TILT0, ROLLc-ROLL0)
[0115] Zoom value = a predetermined value within the range of 0 to 100 [%]
[0116] Return to reference Figure 4 In the volume studio, a plurality of camera devices 71 (camera devices 71-1 to 71-N) installed on the periphery of the studio images a person ACT2 as a subject, and outputs the resulting imaged image to a volume video generating device 72. In order to facilitate distinction between the performer's person ACT2 as a 3D model data generation target and others, the volume studio is green backed.
[0117] The volumetric video generating device 72 generates a 3D model of the person ACT2 from the imaging images supplied from the plurality of cameras 71 by using the volumetric capture technology. The volumetric video generating device 72 outputs the generated 3D model data of the person ACT2 to the volumetric 2D video generating device 22.
[0118] The volumetric 2D video generating device 22 generates a 2D video and a depth video obtained by viewing the 3D model of the person ACT2 from the volumetric video generating device 72 from the virtual camera 73. Here, the volumetric 2D video generating device 22 uses the virtual viewpoint information supplied from the background video generating device 53 as the viewpoint of the virtual camera 73. That is, the volumetric 2D video generating device 22 matches the position, orientation, and zoom value of the virtual camera 73 with the real camera 51, and generates a 2D video and a depth video obtained by viewing the 3D model of the person ACT2 from the viewpoint of the real camera 51. The 2D video and the depth video are a volumetric 2D video (RGB) and a volumetric 2D video (Depth).
[0119] The volumetric 2D video generating device 22 generates a volumetric 2D video (RGB) and a volumetric 2D video (Depth) of the same viewpoint as the real camera device 51 by using the 3D model data of the person ACT2, assigns a background imaging system ID and frame number that are the same as the virtual viewpoint information, and outputs the volumetric 2D video (RGB) and the volumetric 2D video (Depth) to the video synthesizing device 31.
[0120] Figure 6 Examples of data segments generated by the background video generating means 53, the volumetric video generating means 72 and the volumetric 2D video generating means 22 are shown.
[0121] The background video generating means 53 generates background video (RGB) and background video (Depth). A background imaging system ID and a frame number are assigned to the background image (RGB) and the background image (Depth).
[0122] In addition, the virtual viewpoint information generated by the background video generating device 53 includes the position (x, y, z), orientation (pan, tilt, roll) and zoom value of the real camera device 51, the background imaging system ID and the frame number. Figure 6 In the example, the position (x, y, z) of the real camera 51 is (100.0, 1000, 0, 2200, 0), the orientation (pan, tilt, roll) is (-0.1, 10, 0), the zoom value is 50[%], the background imaging system ID is "XXX", and the frame number is "1000".
[0123] The volumetric video generating means 72 generates 3D model data of the person ACT2. The 3D model data of the person ACT2 includes, for example, 3D shape data in a 3D mesh format and texture data in a multi-texture format.
[0124] The volume 2D video generating means 22 generates the volume 2D video (RGB) and the volume 2D video (Depth). The same background imaging system ID and frame number as those of the background video (RGB) and the background video (Depth) are assigned to the volume 2D video (RGB) and the volume 2D video (Depth).
[0125] Figure 7 This is a diagram for describing the composite video generation process performed by the video composite device 31 .
[0126] The video synthesizing device 31 generates a synthesized video (RGB) by synthesizing the background video (RGB) and the background video (Depth) and the volume 2D video (RGB) and the volume 2D video (Depth) having the same background imaging system ID and frame number.
[0127] The video synthesis device 31 sets a predetermined pixel (x, y) of the synthesized video (RGB) to be generated as a pixel of interest, and compares the depth value of the pixel (x, y) of the background video (Depth) corresponding to the pixel of interest with the depth value of the pixel (x, y) of the volumetric 2D video (Depth).
[0128] In the background video (Depth) and the volumetric 2D video (Depth), the magnitude of the depth value is represented by a grayscale value. A larger grayscale value (whiter density) indicates a larger depth value and a shorter distance, and a smaller grayscale value (darker density) indicates a smaller depth value and a longer distance. Figure 7 In the example of the background video (Depth) and the volumetric 2D video (Depth), a depth value indicating that the person ACT2 of the volumetric 2D video (Depth) is closer than the person ACT1 of the background video (Depth) is stored.
[0129] The video synthesis device 31 generates a synthesized video (RGB) to give priority to a closer subject. That is, the video synthesis device 31 selects the RGB value of the pixel (x, y) of the background video (RGB) or the pixel (x, y) of the volume 2D video (RGB) corresponding to the large depth value from the depth value of the pixel (x, y) of the background video (Depth) corresponding to the pixel of interest and the depth value of the pixel (x, y) of the volume 2D video (Depth), and sets the selected RGB value as the RGB value of the pixel (x, y) of the synthesized video (RGB).
[0130] The video synthesizing device 31 generates a synthesized video (RGB) by sequentially setting all pixels constituting the synthesized video (RGB) as pixels of interest and repeating the above-described process of determining the RGB values of the pixels of interest.
[0131] Return to reference Figure 4 , the video synthesis device 31 outputs the generated synthesized video (RGB) to the 2D video transmission device 32. The 2D video transmission device 32 transmits the synthesized video (RGB) to each user's client device 33. The synthesized video (RGB) is also output and displayed on the monitor 12 of the background imaging studio and the monitor 23 of the volume studio.
[0132] Figure 8 Shown are examples of composite videos (RGB) generated at various locations as background imaging studios.
[0133] exist Figure 8 The first composite video (RGB) from the left in the upper part shows an example in which an indoor news studio is imaged as a background imaging studio and a person ACT2 in a volumetric studio is set at the location of the news studio.
[0134] exist Figure 8 The second composite video (RGB) from the upper left shows an example in which an outdoor stadium is imaged as a background imaging studio and a person ACT2 in the volumetric studio is set at the location of the stadium.
[0135] exist Figure 8 The third composite video (RGB) from the upper left (the second composite video (RGB) from the right) shows an example in which an outdoor disaster scene location is imaged as a background imaging studio and a person ACT2 in the volumetric studio is set at the disaster scene.
[0136] exist Figure 8 The first composite video (RGB) from the upper right shows an example in which a person in a volumetric studio ACT2 is set up at a location of a studio in New York (overseas), with the studio in New York serving as a background imaging studio.
[0137] As described above, the person ACT2 in the volumetric studio and the background video imaged by the real camera 51 in the background imaging studio are synthesized, and thus, a 2D video can be generated as if the person ACT2 in the volumetric studio is in front of the real camera 51.
[0138] <4. Volumetric Video Generation Process>
[0139] Next, we will refer to Fig. 9 The flowchart of FIG. 7 describes a volumetric video generation process of generating 3D model data of a person in a volumetric studio performed by the volumetric video generation device 72. For example, when imaging of the cameras 71-1 to 71-N starts and an operation of starting to generate 3D model data is performed in the volumetric video generation device 72, the process starts.
[0140] First, in step S11, the volumetric video generating device 72 acquires imaging images supplied from the N cameras 71, and generates a contour image obtained by expressing the region of a person (subject) as a 3D model generation target object with a contour for each of the imaging images of the camera 71. This processing can be performed by chroma key processing using the green color of a green back as a key signal.
[0141] In step S12, the volumetric video generation device 72 generates (restores) the three-dimensional shape of the object based on the contour image and the camera parameters of the camera 71. More specifically, the volumetric video generation device 72 projects N contour images according to the camera parameters, and generates (restores) the three-dimensional shape of the object by using a visual shell method for cutting out a three-dimensional shape. The three-dimensional shape of the object is represented by voxel data. The camera parameters of the camera 71 are known by calibration.
[0142] In step S13, the volumetric video generation device 72 converts the 3D shape data representing the three-dimensional shape of the object from voxel data into data in a mesh format called a polygonal mesh. For example, an algorithm such as the marching cube method can be used for the conversion of the data format of the polygonal mesh, which is easy to perform rendering processing on a display device.
[0143] In step S14 , the volumetric video generating device 72 performs mesh reduction to merge the number of polygon meshes of the 3D shape data to a target number or less.
[0144] In step S15, the volumetric video generation device 72 generates texture data corresponding to the 3D shape data of the object, and supplies 3D model data including the 3D shape data and texture data of the object to the volumetric 2D video generation device 22. Figure 2 In the case where the multi-texture format described above is used as texture data, the imaged image imaged by each camera 71 is used as texture data as it is. Figure 2 In the case of the described UV mapping format, a UV mapping image corresponding to the shape data of the object is generated as texture data.
[0145] The generated 3D model data of the person in the volume studio is supplied from the volume video generating device 72 to the volume 2D video generating device 22, and Fig. 9 The volume video generation process in is ended. Note that the process is repeated for the imaged images sequentially supplied from the camera 71 as moving images. Fig. 9 Volumetric video generation processing in .
[0146] <5. Volumetric 2D Video Generation Process>
[0147] Next, we will refer to Fig.10 The flowchart of describes the volumetric 2D video generation processing performed by the volumetric 2D video generation device 22 to generate a volumetric 2D video (RGB-D) corresponding to the movement of the real camera device 51. For example, the processing is started when the 3D model data of a person is supplied from the volumetric video generation device 72 and the virtual viewpoint information is supplied from the background video generation device 53.
[0148] First, in step S31, the volume 2D video generating device 22 sets the y coordinate of the pixel of interest (x, y) used to determine the volume 2D video (RGB) and the volume 2D video (Depth) as the output video to 1, and sets the x coordinate to 1 in step S32.
[0149] In step S33 , the volumetric 2D video generating means 22 calculates, for the position (x, y) of the output video, which three-dimensional position of the 3D model of the person in the volumetric studio is drawn based on the virtual viewpoint information from the background video generating means 53 .
[0150] In step S34 , the volumetric 2D video generating device 22 obtains RGB values from the texture data of the 3D model data for the calculated three-dimensional position of the 3D model of the person.
[0151] In step S35 , the volumetric 2D video generation device 22 calculates the distance from the virtual camera device for the calculated three-dimensional position of the 3D model of the person based on the virtual viewpoint information.
[0152] In step S36, the volume 2D video generating device 22 converts the calculated value of the distance from the virtual camera device into a depth value. The conversion from the value of the distance to the depth value can be performed by any method, and for example, the following method can be used. For example, in a case where the imaging area of the volume studio is 10m×10m×3m and the depth value of the volume 2D video (Depth) is represented by 16 bits, since the maximum value of the distance d in the volume studio is 12.65m, 13.0m is set as the maximum value of the distance, and the distance d is converted into a depth value depth by the depth value depth=(65535-d*65535 / 13.0). However, in the absence of a three-dimensional position of a 3D model matching the pixel of interest (x, y), the depth value depth=0. The depth value increases as the distance decreases.
[0153] In step S37, the volume 2D video generating device 22 sets the calculated RGB value and depth value as the volume 2D video (RGB) and the value of the (x, y) position of the volume 2D video (Depth) as the output video. The RGB value obtained from the texture data is set as the value of the (x, y) position of the volume 2D video (RGB), and the depth value converted from the distance value is set as the value of the pixel of interest (x, y) of the volume 2D video (Depth).
[0154] In step S38 , the volumetric 2D video generating device 22 determines whether the value of the x coordinate of the current pixel of interest (x, y) is the same as the width width of the video size of the output video.
[0155] In the case where it is determined in step S38 that the value of the x coordinate of the current pixel of interest (x, y) is not the same as the width width of the video size of the output video, the process proceeds to step S39, and the value of the x coordinate is incremented by 1. Thereafter, the process returns to step S33, and the processes of the above-mentioned steps S33 to S38 are repeated. That is, the process of calculating the value of the (x, y) position of the volume 2D video (RGB) and the volume 2D video (Depth) is performed using another pixel in the same row of the output video as the pixel of interest (x, y).
[0156] At the same time, when it is determined in step S38 that the value of the x coordinate of the current pixel of interest (x, y) is the same as the width width of the video size of the output video, the processing proceeds to step S40, and the volumetric 2D video generating device 22 determines whether the value of the y coordinate of the current pixel of interest (x, y) is the same as the height height of the video size of the output video.
[0157] In the case where it is determined in step S40 that the value of the y coordinate of the current pixel of interest (x, y) is not the same as the height height of the video size of the output video, the process proceeds to step S41, and the value of the y coordinate is incremented by 1. Thereafter, the process returns to step S32, and the processes of the above steps S32 to S40 are repeated. That is, the processes of the above steps S32 to S40 are repeated until all the rows of the output video are set as the pixel of interest (x, y).
[0158] In the case where it is determined in step S40 that the value of the y coordinate of the current pixel of interest (x, y) is the same as the height height of the video size of the output video, the processing proceeds to step S42. In step S42, the volume 2D video generating device 22 assigns the same background imaging system ID and frame number as the virtual viewpoint information to the volume 2D video (RGB) and volume 2D video (Depth) as the generated output video, and outputs the volume 2D video (RGB) and volume 2D video (Depth) to the video synthesizing device 31.
[0159] As mentioned above, Fig.10 The volume 2D video generation process ends. Note that this volume 2D video generation process is also repeatedly performed based on the virtual viewpoint information sequentially supplied from the background video generation device 53.
[0160] <6. Synthetic Video Generation Process>
[0161] Next, we will refer to Fig.11The flowchart of describes the composite video generation process performed by the video synthesis device 31. For example, in the case where the background video (RGB-D) is supplied from the background video generation device 53 and the volumetric 2D video (RGB-D) is supplied from the volumetric 2D video generation device 22, the process is started.
[0162] First, in step S51 , the video synthesizing device 31 sets a variable FN for identifying a frame number to 1.
[0163] In step S52 , the video synthesizing device 31 obtains the background video (RGB) and the background video (Depth) of the frame number FN from the background video generating device 53 .
[0164] In step S53 , the video synthesizing device 31 acquires the volumetric 2D video (RGB) and the volumetric 2D video (Depth) of the frame number FN from the volumetric 2D video generating device 22 .
[0165] In step S54, the video synthesis device 31 performs video synthesis processing to generate a composite video (RGB) so as to give priority to the closer subject. Fig.12 The flowchart describes the details of the video synthesis process.
[0166] In step S55 , the video synthesizing device 31 supplies the generated synthesized video (RGB) to the 2D video transmitting device 32 , and also supplies the synthesized video to the monitor 12 of the background imaging studio and the monitor 23 of the volume studio.
[0167] In step S56 , the video synthesizing device 31 determines whether the video input from the video synthesizing device 31 or the volumetric 2D video generating device 22 is completed.
[0168] In the case where it is determined in step S56 that the video has not been completed, the process proceeds to step S57, and the value of the frame number FN is incremented by 1. Thereafter, the process returns to step S52, and the processes of steps S52 to S56 described above are repeated.
[0169] Meanwhile, in the case where it is determined in step S56 that a video is not supplied from any of the video synthesizing device 31 and the volumetric 2D video generating device 22 and the video is completed, Fig.11 The composite video generation process is completed.
[0170] Fig.12 is shown as Fig.11 Flowchart of details of the video synthesis processing performed in step S54.
[0171] First, the video synthesizing device 31 sets the y coordinate of the pixel of interest (x, y) for determining the synthesized video (RGB) as the output video to 1 in step S71 , and sets the x coordinate to 1 in step S72 .
[0172] In step S73 , the video synthesizing device 31 obtains the depth value depth of the (x, y) position from each of the background video (Depth) and the volumetric 2D video (Depth), converts the depth value depth into a distance d, and selects the depth video with a shorter distance d.
[0173] In step S74, the video synthesizing device 31 obtains the RGB value at the (x, y) position of the RGB video corresponding to the selected depth video. That is, when the selected depth video is the background video (Depth), the video synthesizing device 31 obtains the RGB value at the (x, y) position of the background video (RGB), and when the selected depth video is the volume 2D video (Depth), the video synthesizing device 31 obtains the RGB value at the (x, y) position of the volume 2D video (RGB).
[0174] In step S75 , the video synthesizing device 31 writes the acquired RGB value as the pixel value at the (x, y) position of the synthesized video (RGB) as the output video.
[0175] In step S76 , the video synthesizing device 31 determines whether the value of the x coordinate of the current pixel of interest (x, y) is the same as the width width of the video size of the output video.
[0176] In the case where it is determined in step S76 that the value of the x coordinate of the current pixel of interest (x, y) is not the same as the width width of the video size of the output video, the process proceeds to step S77, and the value of the x coordinate is incremented by 1. Thereafter, the process returns to step S73, and the processes of the above-described steps S73 to S76 are repeated. That is, the process of acquiring and writing an RGB value closer to the distance d is performed using another pixel in the same row of the output video as the pixel of interest (x, y).
[0177] At the same time, when it is determined in step S76 that the value of the x-coordinate of the current pixel of interest (x, y) is the same as the width width of the video size of the output video, the processing proceeds to step S78, and the video synthesis device 31 determines whether the value of the y-coordinate of the current pixel of interest (x, y) is the same as the height height of the video size of the output video.
[0178] In the case where it is determined in step S78 that the value of the y coordinate of the current pixel of interest (x, y) is not the same as the height height of the video size of the output video, the process proceeds to step S79, and the value of the y coordinate is incremented by 1. Thereafter, the process returns to step S72, and the processes of the above steps S72 to S78 are repeated. That is, the processes of the above steps S72 to S78 are repeated until all the lines of the output video are set as the pixel of interest (x, y).
[0179] When it is determined in step S78 that the value of the y coordinate of the current pixel of interest (x, y) is the same as the height height of the video size of the output video, Fig.11 The video synthesis process performed in step S54 ends, and the process proceeds to Fig.11 Step S55.
[0180] As described above, according to the image processing system 1 of the first embodiment, a composite video obtained by synthesizing a video of a subject (e.g., a person) imaged by N cameras 71 in a volume studio and a background video imaged by a real camera 51 in a background imaging studio can be generated in real time, and the composite video can be output to a 2D video transmission device 32.
[0181] <7. Comparison with other systems>
[0182] Fig.13 is a table comparing the image processing system 1 of the first embodiment (hereinafter, referred to as the present system) with another system. Fig.13 Describe the characteristics of this system compared to another system.
[0183] From the perspective of the comparison system, the background creation cost, the background creation period, the authenticity of the background, the naturalness of the foreground and background superposition, the freedom of viewpoint movement, the user's viewpoint movement, and the sense of presence and reality generated in real-time transmission were examined. In addition, "volumetric 3D transmission", "volumetric 2D transmission" and "transmission by chroma key and 2D superposition" are considered to be another system compared with the present system. "Volume 3D transmission" is a method for creating a background with volumetric technology in the following system, in which the virtual viewpoint of the 3D model is determined on the user side of the transmission destination, and the transmission is performed so that each user can freely change the viewpoint. "Volume 2D transmission" is a method for creating a background with volumetric technology in the following system, in which the virtual viewpoint is determined on the transmission side, and the video of the same virtual viewpoint is transmitted to multiple users. "Transmission by chroma key and 2D superposition" is a system in which a 2D video of the background is superimposed on a foreground subject video imaged by a chroma key studio and the superimposed video is transmitted.
[0184] Regarding the background creation cost (cost for creating a background video) and the background creation period (period for creating a background video), "volume 3D transmission" and "volume 2D transmission" using volumetric technology are disadvantageous because the cost and period become large. On the other hand, since "transmission by chroma key and 2D overlay" and the present system immediately use the video actually imaged by the camera, it does not cost and time.
[0185] Regarding the realism of the background (realism of the background video), "Volume 3D transmission" and "Volume 2D transmission" depend on the quality of the 3D CG image of the background. According to "Transmission by chroma key and 2D overlay" and this system, since the background video is a live video actually imaged by the camera device, a high degree of realism can be expressed.
[0186] Regarding the naturalness of the foreground and background superposition, "volumetric 3D transmission", "volumetric 2D transmission" and the present system using a virtual camera (virtual viewpoint information) can express the naturalness of the superposition. On the other hand, in "transmission by chroma key and 2D superposition", it is impossible to make the camera on the foreground side and the camera on the background side accurately overlap with each other, and a method for synthesizing the background video while fixing the viewpoint position is generally performed. Therefore, in "transmission by chroma key and 2D superposition", since the sense of synthesis cannot be eliminated and a natural video cannot be generated, the naturalness of the superposition is reduced.
[0187] Regarding the degree of freedom of viewpoint movement, "Volume 3D transfer" and "Volume 2D transfer" using a virtual camera (virtual viewpoint information) are advantageous. This system can also be moved, but there are certain restrictions on movement due to the use of a real camera 51. Since "Transfer by chroma key and 2D overlay" has a fixed viewpoint position, the degree of freedom is low.
[0188] Regarding the user's viewpoint movement, only "volumetric 3D transfer" can be achieved, and regarding "volumetric 2D transfer", "transfer by chroma key and 2D superposition" and the present system, the user cannot determine the viewpoint.
[0189] Regarding the generation of a sense of presence and reality in real-time transmission, it is difficult to generate a sense of presence and reality in "volumetric 3D transmission" and "volumetric 2D transmission" using 3D CG images as backgrounds. "Transmission by chroma key and 2D superposition" and this system can use a sports stadium or a live venue as a background video, and can generate a high sense of presence and reality.
[0190] As described above, in the present system, compared with "volumetric 3D transmission" and "volumetric 2D transmission", since realistic video of actual imaging can be used as background video at low cost without the need to create a period, and real-time transmission can be performed, users can be provided with a sense of presence and reality at the scene.
[0191] Furthermore, compared to "transmission by chroma key and 2D superposition", since the present system uses the information of the real camera 51 as the virtual camera (virtual viewpoint information), the naturalness of the foreground and background superposition is very high. That is, the performers in the volumetric studio can appear in the background video (live video) in a form that is indistinguishable from the real performers at the venue.
[0192] As described above, according to the present system, 2D transmission capable of producing a sense of reality can be realized at low cost.
[0193] <8. Modification of the First Embodiment>
[0194] Fig.14 is a block diagram showing a modification of the first embodiment of the above-described image processing system.
[0195] In the above-mentioned first embodiment, the background imaging system 11 is installed in the background imaging studio, the volume imaging system 21 and the volume 2D video generating device 22 are installed in the volume studio, the video synthesis device 31 is installed in the video synthesis center, and the 2D video transmission device 32 is installed in the transmission center.
[0196] However, the background imaging system 11, volume imaging system 21, volumetric 2D video generating device 22, video synthesizing device 31 and 2D video transmitting device 32 may not be independently installed at different places, and two or more devices or systems may be installed at the same place.
[0197] For example, Fig.14 As shown, the video synthesis device 31 and the 2D video transmission device 32 may be arranged in a volume studio where the volume imaging system 21 and the volume 2D video generation device 22 are installed.
[0198] On the contrary, although not shown, the video synthesizing device 31 and the 2D video transmitting device 32 may be provided in a background imaging studio in which the background imaging system 11 is installed.
[0199] Alternatively, the video synthesizing device 31 and the 2D video transmitting device 32 may be provided in the same center (eg, a transmission center), and may be provided at three locations of a background imaging studio, a volume studio, and a transmission center.
[0200] <9. Second Embodiment of Image Processing System>
[0201] Next, a second embodiment of the image processing system to which the present technology is applied will be described. The second embodiment is a mode in which a plurality of background imaging systems 11 or a plurality of volume imaging systems 21 are provided.
[0202] In the drawings of the second embodiment to be described later, Figure 3 Corresponding parts of the illustrated first embodiment are denoted by the same reference numerals, and descriptions of these parts will be appropriately omitted to focus on the description of different parts.
[0203] <First Configuration Example of Second Embodiment>
[0204] Fig.15 is a block diagram showing a first configuration example of the image processing system according to the second embodiment.
[0205] The first configuration example of the second embodiment is a configuration in which a plurality of background imaging systems 11 are provided in one background imaging studio.
[0206] exist Fig.15 In the embodiment, two background imaging systems 11 are provided in one background imaging studio. In addition, two monitors 12 are provided to correspond to the two background imaging systems 11. Note that Fig.15 The example is an example of a case where two background imaging systems 11 are provided, but three or more background imaging systems 11 may be provided, of course.
[0207] Furthermore, in the above-described first embodiment, although the camera of the background imaging studio includes the RGB camera 51R and the depth camera 51D, in the second embodiment, the camera includes a stereo camera 54. Since the camera of the background imaging system 11 may be required only to acquire the RGB value (2D video) and the depth value (depth video) of the subject, the camera may be the stereo camera 54, rather than a combination of the RGB camera 51R and the depth camera 51D. The stereo camera 54 generates the RGB value (2D video) and the depth value (depth video) of the subject by performing stereo matching processing based on two RGB videos obtained by imaging the subject.
[0208] Furthermore, in the first configuration example of the second embodiment, although a plurality of background imaging systems 11 are provided, the same number (two) of volume 2D video generating devices 22 and monitors 23 of the volume studio as the background imaging systems 11 and a video synthesizing device 31 of the video synthesizing center are provided.
[0209] In other words, in the first configuration example of the second embodiment, the monitor 12 of the background imaging studio, the volumetric 2D video generating device 22 and the monitor 23 of the volumetric studio, and the video synthesizing device 31 of the video synthesizing center are provided to correspond to one background imaging system 11. The synthesized video (RGB) corresponding to the background video (RGB-D) imaged by the stereo camera device 54 is generated by the set of the volumetric 2D video generating device 22 and the video synthesizing device 31 corresponding to the background imaging system 11.
[0210] Furthermore, in the transmission center, a switcher (selection unit) 81 and a composite video selection device 82 are added to the front stage of the 2D video transmission device 32. The switcher 81 generates a monitoring video in which the composite videos (RGB) supplied from the two video synthesis devices 31 are integrated into one screen, and supplies the monitoring video to the composite video selection device 82. Furthermore, the switcher 81 selects one of the composite videos (RGB) supplied from the two video synthesis devices 31 based on the transmission video selection instruction supplied from the composite video selection device 82, and supplies the selected composite video to the 2D video transmission device 32.
[0211] The composite video selection device 82 displays the monitoring video supplied from the switch 81 on the external display. The composite video selection device 82 generates a transmission video selection instruction for selecting one of the two composite videos (RGB) included in the monitoring video based on the selection operation of the user (operator) for confirming the monitoring video displayed on the external display, and supplies the transmission video selection instruction to the switch 81. The user (operator) who operates the composite video selection device 82 confirms the monitoring video, and performs a button operation of selecting one of the two composite videos (RGB) included in the monitoring video as the transmission video. In addition, the composite video selection device 82 may be able to specify how to integrate the two composite videos (RGB) into one screen as the monitoring video.
[0212] The first configuration example of the second embodiment has the above configuration.
[0213] Note that, in the second exemplary embodiment, although the switch 81 selects one of the composite videos (RGB) supplied from the two video composite devices 31 and supplies the selected composite video to the 2D video transmission device 32, the composite videos (RGB) supplied from the two video composite devices 31 may be arranged horizontally, or a composite video integrated into one screen by setting the screen size to be different by PinP (Picture in Picture) or the like may be generated and may be supplied as a transmission video to the 2D video transmission device 32. In this case, the composite video selection device 82 may be omitted.
[0214] In the case where three or more background imaging systems 11 are provided in the background imaging studio, the volume 2D video generating devices 22 of the volume studio and the video synthesizing device 31 of the video synthesizing center are provided in the same number as the background imaging systems 11. The switch 81 selects one of a plurality of synthesized images (RGB) and supplies the selected synthesized image to the 2D video transmitting device 32.
[0215] Will refer to Fig.16 The operation of the first configuration example in the second embodiment is described.
[0216] In the background imaging studio, two stereo cameras 54 image a person ACT1 as a subject at different camera positions and orientations. Each of the two stereo cameras 54 outputs the 2D video and depth video obtained by imaging the subject to the corresponding background video generating device 53. Similar to the first embodiment, the camera motion detection sensor 52 also acquires the position, orientation, and zoom value of the corresponding stereo camera 54, and outputs the acquired position, orientation, and zoom value to the corresponding background video generating device 53.
[0217] Each of the two background video generating devices 53 sets the 2D video and depth video having the same viewing angle supplied from the corresponding stereo camera device 54 as the background video (RGB) and the background video (Depth), assigns a background imaging system ID and a frame number, and outputs the background video to the video synthesizing device 31. In addition, the background video generating device 53 assigns the background imaging system ID and the frame number to the position, orientation, and zoom value of the corresponding stereo camera device 54, and outputs the position, orientation, and zoom value as virtual viewpoint information to the corresponding volumetric 2D video generating device 22.
[0218] Each of the volumetric 2D video generating devices 22 generates a volumetric 2D video (RGB) and a volumetric 2D video (Depth) of the person ACT2 from the same viewpoint as the corresponding stereo camera device 54 by using the virtual viewpoint information supplied from the corresponding background video generating device 53, and outputs the generated image to the corresponding video synthesizing device 31. That is, the volumetric 2D video generating device 22 assumes that the virtual camera device 73 moves in the same manner as the corresponding stereo camera device 54, and generates a volumetric 2D video (RGB) and a volumetric 2D video (Depth) viewed from the virtual camera device 73.
[0219] Each of the two video synthesis devices 31 generates a synthesized video (RGB) by synthesizing the background video (RGB) and background video (Depth) supplied from the corresponding background video generation device 53 and the volumetric 2D video (RGB) and volumetric 2D video (Depth) supplied from the corresponding volumetric 2D video generation device 22 having the same background imaging system ID and frame number.
[0220] The switch 81 selects one of the two composite videos (RGB) and supplies the selected composite video to the 2D video transmission device 32. The transmission video transmitted to the client device 33 is a video that simulates the situation where there are two camera devices in the studio. That is, the video is the same background video in which two composite videos (RGB) with different imaging angles are switched.
[0221] <Second Configuration Example of Second Embodiment>
[0222] Fig.17 is a block diagram showing a second configuration example of the image processing system according to the second embodiment.
[0223] The second configuration example of the second embodiment is a configuration in which two background imaging studios are provided and one background imaging system 11 is provided in each of the background imaging studios.
[0224] exist Fig.17 In spite of Fig.15 The first configuration example similarly provides two background imaging systems 11 and two monitors 12, but the two background imaging systems and the two monitors differ from each other in that the two background imaging systems and the two monitors are set one by one in a background imaging studio rather than being set in one background imaging studio.
[0225] Therefore, in the first configuration example, although the two composite videos (RGB) supplied to the switch 81 are the same background video (however, the imaging angles are different), the two composite videos (RGB) of the second configuration example are different background videos.
[0226] Fig.18 An example of a transmission video in the case where four background imaging studios are provided and one background imaging system 11 is provided in each background imaging studio in the second configuration example of the second embodiment is shown.
[0227] There are four background imaging locations, namely, an indoor news studio, an outdoor stadium, an outdoor disaster site, and New York (overseas), and a background imaging system 11 is installed in each of the four background imaging locations.
[0228] There is a volume studio, and a volume imaging system 21 images a person ACT2.
[0229] In this case, the switch 81 sequentially selects and switches the four synthesized videos (RGB) supplied from the four video synthesizing devices 31 as transmission videos, and thus, it is possible to transmit a scene in which the person ACT2 instantaneously moves and participates at each background imaging location.
[0230] <Third Configuration Example of Second Embodiment>
[0231] Fig.19 is a block diagram showing a third configuration example of the image processing system according to the second embodiment.
[0232] The third configuration example of the second embodiment is a configuration in which two volume studios are provided and in each volume studio a volume imaging system 21, a volume 2D video generating device 22 and a monitor 23 are provided. For distinction, the two volume studios are referred to as volume studio A and volume studio B.
[0233] In the video synthesis center, the same number (two) of video synthesis devices 31 as the volumetric 2D video generation device 22 are provided, and a switcher 81 and a synthesized video selection device 82 are added at the preceding stage of the 2D video transmission device 32 in the transmission center.
[0234] The background video generating device 53 of the background imaging studio assigns the background imaging system ID and the frame number to the generated background video (RGB-D), and outputs the background video to the plurality of video synthesizing devices 31. In addition, the background video generating device 53 generates virtual viewpoint information, and outputs the virtual viewpoint information to the volume 2D video generating device 22 of each volume studio. In the background imaging studio, two monitors 12 are installed to display the synthesized video (RGB) generated by the two video synthesizing devices 31.
[0235] The volume imaging system 21 of the volume studio A generates a 3D model of a person ACT2 as a subject, and outputs the 3D model data to the corresponding volume 2D video generating device 22 .
[0236] The volumetric 2D video generating device 22 of volumetric studio A generates a volumetric 2D video (RGB-D) of person ACT2 from the same viewpoint as the stereoscopic camera device 54 by using the 3D model data of person ACT2, assigns a background imaging system ID and frame number that are the same as the virtual viewpoint information, and outputs the generated video to the corresponding video synthesizing device 31 (first video synthesizing device 31).
[0237] The volume imaging system 21 of the volume studio B generates a 3D model of the person ACT3 as a subject, and outputs the 3D model data to the corresponding volume 2D video generating device 22 .
[0238] The volumetric 2D video generating device 22 of the volumetric studio B generates a volumetric 2D video (RGB-D) of the person ACT3 having the same viewpoint as the stereoscopic camera device 54 by using the 3D model data of the person ACT3, assigns a background imaging system ID and frame number that are the same as the virtual viewpoint information, and outputs the generated video to the corresponding video synthesizing device 31 (second video synthesizing device 31).
[0239] The first video synthesizing means 31 acquires the background video (RGB-D) from the background video generating means 53 and acquires the volume 2D video (RGB-D) from the volume 2D video generating means 22 of the volume studio A, and generates a synthesized video (RGB) of the person ACT2.
[0240] The second video synthesizing means 31 acquires the background video (RGB-D) from the background video generating means 53 and acquires the volume 2D video (RGB-D) from the volume 2D video generating means 22 of the volume studio B, and generates a synthesized video (RGB) of the person ACT3.
[0241] The switch 81 generates a monitoring video in which two composite videos (RGB) supplied from the two video synthesizing devices 31 are integrated into one screen, and supplies the generated monitoring video to the composite video selecting device 82. In addition, the switch 81 selects one of the composite video (RGB) of the person ACT2 or the composite video (RGB) of the person ACT3 based on the transmission video selection instruction supplied from the composite video selecting device 82, and supplies the selected composite video to the 2D video transmission device 32.
[0242] As described above, in the third configuration example, the image processing system 1 can generate the following two composite videos, and can select and transmit one of the composite videos: a composite video (RGB) obtained by synthesizing a person ACT2 in a volume studio A and a composite video (RGB) obtained by synthesizing a person ACT3 in a volume studio B and the same background video (RGB-D).
[0243] Alternatively, in the third configuration example, as Fig. 20As shown, the image processing system 1 can also generate and transmit a composite video (RGB) obtained by synthesizing the person ACT2 and the person ACT3 in different volume studios into one screen. In this case, the volumetric 2D video (RGB-D) of the person ACT2 and the volumetric 2D video (RGB-D) of the person ACT3 are supplied to a video synthesizing device 31. The video synthesizing device 31 compares the depth values at the same pixel position of the background video (RGB-D), the volumetric 2D video (RGB-D) of the person ACT2, and the volumetric 2D video (RGB-D) of the person ACT3, and generates a composite video (RGB).
[0244] As described above, a plurality of background imaging systems 11 may be provided in one background imaging studio, a plurality of background imaging systems 11 may be provided in a plurality of background imaging studios, or a plurality of volume imaging systems 21 may be provided in a plurality of volume studios.
[0245] Note that although not shown, multiple background imaging studios and multiple volume studios may also be provided.
[0246] In the second embodiment, although the stereo camera 54 is used instead of the RGB camera 51R and the depth camera 51D, it goes without saying that the RGB camera 51R and the depth camera 51D can be used similarly to the first embodiment. In addition, in the first embodiment and other embodiments as described later, the RGB camera 51R and the depth camera 51D and the stereo camera 54 can be replaced with each other.
[0247] <10. Third Embodiment of Image Processing System>
[0248] Next, a third embodiment of the image processing system will be described.
[0249] In the third embodiment, it is assumed that the location of the background imaging system 11 is in an outdoor imaging environment called a location site rather than in a studio (background imaging studio) in a general building. For example, the background imaging system 11 performs imaging while moving in a travel program, a marathon roadside broadcast, a broadcast such as mountain climbing, or any location (location) such as a town. In such a case, unlike a studio in a building, since the imaging range moves, the image processing system 1 has a configuration in which the position of the origin can move with the movement of the real camera 51.
[0250] Fig.21 This is a block diagram showing a third embodiment of an image processing system to which the present technology is applied.
[0251] In the drawings of the third embodiment, Figure 3Corresponding parts of the illustrated first embodiment are denoted by the same reference numerals, and descriptions of these parts will be appropriately omitted to focus on the description of different parts.
[0252] In the third embodiment, the background imaging system 11 is set at the location site (location venue). In addition to the camera 51R, camera 51D, camera motion detection sensor 52 and background video generating device 53 similar to the first embodiment, the background imaging system 11 also includes a mode selection button 55 and an origin position designation button 56. In the third embodiment, the camera 51R and the camera 51D (real camera 51) are, for example, small cameras that are easy to move and image. The mode selection button 55 and the origin position designation button 56 can be set as operation buttons of the camera 51R and the camera 51D.
[0253] The camera motion detection sensor 52 includes a sensor suitable for the real camera 51 to be moved, such as a global positioning system (GPS), a gyro sensor, or an acceleration sensor.
[0254] The mode selection button 55 is a button for the camera operator to control whether to move the origin position. The camera operator can switch between three coordinate setting modes: link mode, lock mode, and calibration mode by operating the mode selection button 55.
[0255] The link mode is a mode in which the origin position moves in conjunction with the movement of the real camera 51. In the link mode, when the real camera 51 moves, the origin position on the location scene (background imaging studio) moves as it is, but the virtual viewpoint position indicated as the relative coordinates relative to the origin does not move.
[0256] In the lock mode, the origin position is locked to the currently set position. After the origin position is fixed at a predetermined location, the lock mode operates similarly to the first embodiment (fixed camera). That is, the difference in the movement of the real camera 51 is processed as the movement amount of the virtual camera as it is. The imaging range is fixed, and the virtual viewpoint moves according to the movement of the camera operator.
[0257] The correction mode operates basically the same as the link mode, but the camera operator can freely correct (move) the origin position. For example, in the case where the height of the floor surface is different between the origin position and the position of the camera operator, in the link mode, the position of the performer displayed in the composite video (RGB) may float or may be stuck on the floor surface. Such a shift can be corrected by the camera operator correcting the origin position in the correction mode.
[0258] The origin position designation button 56 is a button for designating the corrected origin position when the origin position is corrected in the correction mode. Any designation method can be used as long as the correction value (movement amount) of each of the x-coordinate, y-coordinate, and z-coordinate can be specified. In addition, not only the origin position can be corrected, but also the orientation of the real camera 51 can be corrected.
[0259] In a third embodiment, Fig.14 The modification of the first embodiment shown is similar, with the video synthesis device 31 and the 2D video transmission device 32 being arranged in the volume studio.
[0260] The other configuration of the third embodiment is similar to that of the above-described first embodiment. The image processing system 1 of the third embodiment has the above configuration.
[0261] Will refer to Fig. 22 and Fig.23 Describes the origin operation in each coordinate setting mode of link mode, lock mode, and correction mode.
[0262] Fig. 22 is an image diagram showing a scene imaged by the background imaging system 11 and the volume imaging system 21 according to the third embodiment.
[0263] The location scene is a passageway in which buildings are arranged left and right. The real camera 51 images a person ACT1 standing on the passageway. At the same time, a person ACT2 appears in the volume studio as a performer, and the volume imaging system 21 is imaging the person ACT2.
[0264] The composite video (RGB) obtained by synthesizing the background video (RGB-D) obtained by imaging the person ACT1 at the location scene and the volumetric 2D video (RGB-D) obtained by imaging the person ACT2 in the volumetric studio is a video as if both the person ACT1 and the person ACT2 are present in the channel at the location scene.
[0265] Fig. 22 The plan view on the right side shows the positional relationship between the passage and the person ACT1 and the moving direction of the person ACT1. The person ACT1 moves in the depth direction in the passage, and the real camera 51 performs imaging while moving according to the movement of the person ACT1.
[0266] Fig.23 The origin processing in each coordinate setting mode of the link mode, the lock mode, and the correction mode is shown.
[0267] In the link mode, when the real camera 51 moves with the movement of the person ACT1, the origin position and the imaging target range of the location scene also move. The virtual viewpoint position shown as the relative coordinates relative to the origin does not move. The composite video (RGB) is a video in which the person ACT1 and the person ACT2 are at the same position in the screen and the building in the background moves with the movement of the person.
[0268] In the lock mode, even if the real camera 51 moves, the origin position and the imaging target range do not move. The movement of the real camera 51 is processed as the movement of the virtual camera as it is. For example, in the case where it is desired to fix the imaging target range, the imaging target range is suitable for imaging when the person ACT1 stops.
[0269] The operation of the correction mode is basically similar to that of the link mode. The camera operator can correct (move) the origin position by operating the origin position designation button 56.
[0270] Fig.24 A control example of the camera position of virtual viewpoint information in each coordinate setting mode is shown.
[0271] because Fig.24 The normal mode on the left is Figure 5 This is an example of the virtual viewpoint information of the first embodiment described in , and thus a description thereof will be omitted.
[0272] In the link mode, even if the real camera 51 moves, the camera position (x, y, z) of the virtual viewpoint information does not change. The absolute coordinates of the origin position (X0, Y0, Z0) move.
[0273] In the locked mode, the absolute coordinates of the origin (X0, Y0, Z0) are fixed to the origin position at the start time of the locked mode. When the real camera 51 moves by the movement amount (dx, dy, dz), the camera position of the virtual viewpoint information is (x+dx, y+dy, z+dz), and reflects the movement amount from the start time of the locked mode.
[0274] In the correction mode, basically, even if the real camera 51 moves similarly to the link mode, the camera position (x, y, z) of the virtual viewpoint information does not change. However, the origin position (X0, Y0, Z0) can be corrected (moved).
[0275] Fig.25 An example of a mode selection button 55 and an origin position designation button 56 is shown.
[0276] exist Fig.25In the example of , the mode selection button 55 and the origin position designation button 56 are provided as a touch panel in the monitor 51M of the real camera 51. Whenever the mode selection button 55 is pressed, the display of the link mode, the lock mode, and the correction mode is switched. The origin position designation button 56 includes a total of six movement buttons that move in the positive direction and the negative direction relative to each of the x-axis, the y-axis, and the z-axis.
[0277] A screen 101 shows an example of a screen displayed on the monitor 51M of the real camera 51 in the link mode. In the link mode, since the origin position cannot be moved, the origin position designation button 56 is controlled to be inoperable.
[0278] A screen 102 shows an example of a screen displayed on the monitor 51M of the real camera 51 in the lock mode. In the lock mode, since the origin position cannot be moved, the origin position designation button 56 is controlled to be inoperable.
[0279] A screen 103 shows an example of a screen displayed on the monitor 51M of the real camera 51 in the calibration mode. In the calibration mode, since the origin position can be moved, the origin position designation button 56 is controlled to be operable.
[0280] The synthesized video (RGB) supplied from the video synthesizing device 31 in the video synthesizing center is displayed on the video display unit 111 of the screens 101 to 103 .
[0281] <Virtual Viewpoint Information Generation Processing>
[0282] Will refer to Fig.26 The flowchart of FIG. 1 describes the virtual viewpoint information generation processing performed by the background video generation device 53 of the third embodiment. This processing starts with the start of imaging in the background imaging system 11, for example.
[0283] First, in step S91, the background video generating means 53 acquires sensor values from the camera motion detecting sensor 52 including a GPS, a gyro sensor, an acceleration sensor, etc., and acquires the absolute coordinates (Xc, Yc, Zc) of the camera position.
[0284] In step S92, the background video generating means 53 determines whether the current coordinate setting mode is the lock mode. In the case where it is determined in step S92 that the current coordinate setting mode is the lock mode, the process proceeds to step S98, as described later.
[0285] Meanwhile, in the case where it is determined in step S92 that the current coordinate setting mode is not the lock mode, the process proceeds to step S93, and the background video generating means 53 calculates the difference (dx, dy, dz) from the previous absolute coordinates. Subsequently, in step S94, the background video generating means 53 sets the origin position (X0, Y0, Z0) to (X0+dx, Y0+dy, Z0+dz) by using the calculated difference (dx, dy, dz).
[0286] Next, in step S95, the background video generating means 53 determines whether the current coordinate setting mode is the link mode. In the case where it is determined in step S95 that the current coordinate setting mode is the link mode, the process proceeds to step S98, as described later.
[0287] Meanwhile, in the case where it is determined in step S95 that the current coordinate setting mode is not the link mode, the processing proceeds to step S96, and the background video generating means 53 determines whether there is no correction value, that is, whether the origin position designation button 56 is operated. In the case where it is determined in step S96 that the origin position designation button 56 is not operated and there is no correction value, the processing proceeds to step S98, as described later.
[0288] Meanwhile, in the case where it is determined in step S96 that the origin position designation button 56 is operated and the origin position is designated by the user, the process proceeds to step S97, and the background video generating means 53 corrects the origin position (X0, Y0, Z0) based on the value (ux, uy, uz) designated by the user. The value (ux, uy, uz) designated by the user is the position designated by the user using the origin position designation button 56, and the corrected origin position (X0, Y0, Z0) is (X0+ux, Y0+uy, Z0+uz).
[0289] In step S98, the background video generating device 53 calculates the virtual viewpoint position based on the camera position (Xc, Yc, Zc) and outputs the virtual viewpoint information. When the position of the real camera 51 is (Xc, Yc, Zc), the camera position (x, y, z) as the virtual viewpoint information is calculated by (Xc-X0, Yc-Y0, Zc-Z0). Then, the background video generating device 53 outputs the calculated virtual viewpoint position (Xc-X0, Yc-Y0, Zc-Z0) together with the orientation (pan, tilt, roll) and zoom value of the real camera 51 as virtual viewpoint information to the volume 2D video generating device 22.
[0290] therefore, Fig.26 The virtual viewpoint information generation process in is completed. This process is repeated periodically at a predetermined cycle.
[0291] According to the third embodiment of the above-mentioned image processing system 1, the origin position can be moved according to the movement of the real camera device 51 at the location scene, etc. Therefore, when imaging the background video (RGB-D) while moving the real camera device 51, a natural synthetic video (RGB) can be generated using the volumetric 2D video (RGB-D) of the volumetric studio.
[0292] <Example of a smartphone or drone>
[0293] In the third embodiment, a full-blown background imaging camera equivalent to that used in a background imaging studio may be used, but a smartphone camera may be used as a real camera 51 used in an outdoor environment. Furthermore, a camera of a drone capable of imaging from above may be used.
[0294] Fig. 27 There is shown a configuration example of a camera motion detection sensor 52 , a mode selection button 55 , an origin position designation button 56 , and the like in a case where a smartphone or a drone is used as the real camera 51 .
[0295] Fig. 27 The left side of shows the appearance of the smartphone 141 and a display screen example of the display 144 of the smartphone 141.
[0296] The camera 142 provided on the rear surface side of the smartphone 141 is used as a real camera 51 that images a subject and generates background video (RGB) and background video (Depth). The body of the smartphone 141 includes a sensor unit 143 such as a GPS, a gyro sensor, and an acceleration sensor, and the sensor unit 143 is used as a camera motion detection sensor 52.
[0297] A mode selection button 55 , an origin position designation button 56 , a video switching button 145 , a video display unit 146 , and the like are provided on the display 144 of the smartphone 141 .
[0298] The video switching button 145 is a button for switching the video displayed on the video display unit 146. In the video display unit 146, the video imaged by the camera 142 and the synthesized video (RGB) supplied from the video synthesizing device 31 can be alternately switched every time the video switching button 145 is pressed. The video display unit 146 displays one of the video imaged by the camera 142 and the synthesized video (RGB) supplied from the video synthesizing device 31 according to the setting state of the video switching button 145.
[0299] The smartphone is used as the real camera 51, and therefore, even in an environment where a full-fledged background imaging camera cannot be prepared, a synthetic video (RGB) creating a sense of presence can be transmitted by using live video of the imaging location in real time.
[0300] exist Fig. 27 On the right side of FIG. 1 , an example of a drone 151 and a controller 154 for operating the drone 151 is shown.
[0301] The camera 152 provided on a predetermined surface of the drone 151 is used as a real camera 51 that images a subject and generates background video (RGB) and background video (Depth). The body of the drone 151 includes a sensor unit 153 such as a GPS, a gyro sensor, and an acceleration sensor, and the sensor unit 153 is used as a camera motion detection sensor 52.
[0302] Joysticks 155R and 155L and a display 156 are provided in the controller 154. The joysticks 155R and 155L are operation units for controlling the movement of the drone 151. On the display 156, a mode selection button 55, an origin position designation button 56, a video switching button 157, a video display unit 158, and the like are provided.
[0303] The video switching button 157 is a button for switching the video displayed on the video display unit 158. In the video display unit 158, the video imaged by the camera 152 and the synthesized video (RGB) supplied from the video synthesizing device 31 can be alternately switched every time the video switching button 157 is pressed. The video display unit 158 displays one of the video imaged by the camera 152 and the synthesized video (RGB) supplied from the video synthesizing device 31 according to the setting state of the video switching button 157.
[0304] The camera 152 of the drone 151 is used as the real camera 51, and therefore, even in a place or environment where the real camera 51 cannot be set up, a synthetic video (RGB) that creates a sense of presence can be transmitted by using the live video of the imaging place in real time.
[0305] <11. Fourth Embodiment of Image Processing System>
[0306] Next, a fourth embodiment of the image processing system will be described.
[0307] In the fourth embodiment, the image processing system 1 has a configuration in which the lighting environment of the background imaging studio or the location site where background imaging is performed can be reflected in the volume studio.
[0308] Fig.28This is a block diagram showing a fourth embodiment of an image processing system to which the present technology is applied.
[0309] In the drawings of the fourth embodiment, Figure 3 Corresponding parts of the illustrated first embodiment are denoted by the same reference numerals, and descriptions of these parts will be appropriately omitted to focus on the description of different parts.
[0310] In the fourth embodiment, in addition to the camera 51R, the camera 51D, the camera motion detection sensor 52 and the background video generation device 53 similar to those in the first embodiment, an illumination sensor 57 is provided in the background imaging system 11 .
[0311] The lighting sensor 57 includes a plurality of illuminance sensors, acquires illuminance values around 360°, and supplies the acquired illuminance values to the background video generating device 53. The illuminance value is, for example, a value within a range of 0 to 100%.
[0312] The background video generating device 53 acquires the illuminance value of each of the plurality of illuminance sensors supplied from the illumination sensor 57 , and supplies the illuminance value together with the virtual viewpoint information as illuminance information to the volumetric 2D video generating device 22 .
[0313] In the volume studio, there are additionally provided a lighting control device 181 and a plurality of lighting devices 182. Furthermore, a video synthesis device 31 and a 2D video transmission device 32 are also provided in the volume studio.
[0314] The volumetric 2D video generating device 22 supplies the illuminance information supplied from the background video generating device 53 of the background imaging system 11 to the lighting control device 181 .
[0315] The lighting control device 181 generates lighting control information for controlling the plurality of lighting devices 182 based on the illumination information from the volumetric 2D video generating device 22, and supplies the lighting control information to each of the plurality of lighting devices 182. The number and position of the illumination sensors included in the illumination sensor 57 may not be consistent with the number and position of the lighting devices 182 installed in the volumetric studio. The lighting control device 181 generates lighting control information for each of the plurality of lighting devices 182 installed in the volumetric studio based on the acquired illumination information to reproduce the lighting environment of the background imaging studio. The lighting control information is a control signal for controlling the light emission brightness when the lighting device 182 emits light, and the lighting device 182 emits light at a predetermined light emission brightness based on the lighting control information from the lighting control device 181.
[0316] The other configuration of the fourth embodiment is similar to that of the above-described first embodiment. The image processing system 1 of the fourth embodiment has the above configuration.
[0317] Will refer to Fig.29 The illuminance information acquired by the illumination sensor 57 and the illumination control information for controlling the illumination device 182 are described.
[0318] For example, Fig.29 As shown, the illumination sensor 57 is installed beside the real camera 51 in the background imaging studio or location scene. In the illumination sensor 57, a plurality of illuminance sensors 201 are arranged at the upper, interrupted and lower sections of a substantially spherical shape, so that illumination intensity from various orientations can be measured.
[0319] Each illuminance sensor 201 of the lighting sensor 57 outputs (sensor number, pan, tilt, brightness) as illuminance information to the background video generating device 53. "Sensor number" indicates an identification number for identifying the illuminance sensor 201, "pan" indicates the orientation of the illuminance sensor 201 in the left-right direction, and "tilt" indicates the orientation of the illuminance sensor 201 in the up-down direction. Brightness indicates the illuminance value detected by the illuminance sensor 201.
[0320] In this example, for simplicity, the number of illuminance sensors 201 provided in the illumination sensor 57 is K (K>0), and the number of illumination devices 182 installed in the volume studio is also K, which is the same as the number of illuminance sensors 201. In addition, among the K illumination devices 182, the orientation of the illumination device 182 having the same illumination number as the sensor number corresponds to the orientation of the illuminance sensor 201 having the same sensor number. In other words, each illumination device 182 has (illumination number, pan, tilt) as illumination information, and "pan" and "tilt" of the illumination device 182 having an illumination number of k (k=an integer from 1 to K) are the same as "pan" and "tilt" of the illuminance sensor 201 having a sensor number of k. In this case, the illumination control device 181 can generate the illumination control information of the illumination device 182 having an illumination number of k based on the "brightness" of the illuminance information of the illuminance sensor 201 having a sensor number of k.
[0321] Note that, in the case where the number and orientation of the illuminance sensors 201 are different from the number and orientation of the lighting devices 182 , the lighting control information can be obtained by analyzing using the illuminance information of each illuminance sensor 201 and the lighting information of each lighting device 182 .
[0322] <Lighting Control Processing>
[0323] Next, we will refer to Fig.30 The flowchart of FIG. 1 describes the illumination control process performed by the image processing system 1 of the fourth embodiment. This process starts with the start of imaging in the background imaging system 11, for example.
[0324] First, in step S121, the background video generating device 53 of the background imaging system 11 acquires a plurality of pieces of lighting information of K illumination sensors supplied from the illumination sensor 57. The background video generating device 53 supplies the illumination value of each of the K illumination sensors together with the virtual viewpoint information as illumination information to the volumetric 2D video generating device 22. The lighting information supplied to the volumetric 2D video generating device 22 is output to the lighting control device 181.
[0325] In step S122 , the lighting control device 181 substitutes 1 into a variable k for identifying a lighting number.
[0326] In step S123 , the lighting control device 181 acquires “brightness” from the illuminance information of the illuminance sensor 201 having the sensor number k.
[0327] In step S124 , the lighting control device 181 generates lighting control information of the lighting device 182 having the lighting number k based on the “brightness” of the illuminance information of the illuminance sensor 201 having the sensor number k.
[0328] In step S125, the lighting control device 181 outputs the generated lighting control information to the lighting device 182 with the lighting number k. Therefore, the lighting device 182 with the lighting number k emits light at a predetermined light emission brightness based on the lighting control information.
[0329] In step S126, the lighting control device 181 determines whether the variable k is equal to the number K of the lighting devices 182. In the case where it is determined in step S126 that the variable k is not equal to the number K of the lighting devices 182, in other words, the variable k is smaller than the number K, the process proceeds to step S127, and the variable k is incremented by 1. Then, the process returns to step S123, and the processes of steps S123 to S126 described above are performed on the next lighting device 182.
[0330] Meanwhile, in the case where it is determined in step S126 that the variable k is equal to the number K of lighting devices 182, Fig.30 The lighting control process ends.
[0331] Fig.30 The lighting control process of corresponds to the process of performing light emission control once on K lighting devices 182. Repeated execution Fig.30 The lighting control process is performed until the imaging in the background imaging system 11 is completed.
[0332] According to the fourth embodiment of the above-mentioned image processing system 1, the lighting environment of the background imaging studio or the location scene where the background imaging is performed is reflected in the volume studio. Therefore, a volumetric 2D video (RGB-D) reflecting the lighting environment of the background imaging studio or the location scene can be generated.
[0333] <12. Fifth Embodiment of Image Processing System>
[0334] Next, a fifth embodiment of the image processing system will be described.
[0335] In the first embodiment described above, in order to facilitate the distinction between the performer's person ACT2 as the 3D model data generation target and others, the volume studio is green-backed. However, in the green-backed environment, the performer cannot feel the realism of the scene and it is difficult to perform. Therefore, in the fifth embodiment, an image processing system 1 is formed so that the performer in the volume studio can feel the realism of the scene by setting the wall display around the person ACT2 in the volume studio and projecting the video of the background imaging studio or the location scene on the wall display.
[0336] Fig.31 This is a block diagram showing a fifth embodiment of an image processing system to which the present technology is applied.
[0337] In the drawings of the fifth embodiment, Figure 3 Corresponding parts of the illustrated first embodiment are denoted by the same reference numerals, and descriptions of these parts will be appropriately omitted to focus on the description of different parts.
[0338] In the fifth embodiment, in addition to the camera 51R, camera 51D, camera motion detection sensor 52 and background video generating device 53 similar to those in the first embodiment, an omnidirectional camera 58 is provided in the background imaging system 11. The omnidirectional camera 58 is a camera that images omnidirectional video, and is used to enable the performers in the volumetric studio to grasp the situation in the background imaging studio or location scene with a sense of reality.
[0339] Meanwhile, an omnidirectional video output device 221 and a plurality of (K) wall displays 222-1 to 222-K are additionally provided in the volume studio. In addition, a video synthesis device 31 and a 2D video transmission device 32 are also provided in the volume studio.
[0340] Fig.32 An example of a setup of a real camera 51 and an omnidirectional camera 58 in a background imaging studio or location site and K wall displays 222 - 1 to 222 -K in a volume studio is shown.
[0341] For example, Fig.32 As shown, the omnidirectional camera 58 is arranged so that its viewing angle does not fall within the viewing angle of the real camera 51, for example, the upper part of the real camera 51. The omnidirectional camera 58 images the periphery around the real camera 51, and supplies the omnidirectional image obtained as a result of the imaging to the background video generating device 53. The omnidirectional camera 58 may be arranged at a place different from the real camera 51.
[0342] K wall displays 222 - 1 to 222 -K are arranged around the origin to surround the volume studio. Fig.32 The example of FIG. 1 shows an example in which eight wall displays 222 - 1 to 222 - 8 are arranged around the volume studio, where K=8.
[0343] Return to reference Fig.31 As described above, the background video generating device 53 supplies the omnidirectional video supplied from the omnidirectional camera device 58 to the omnidirectional video output device 221 of the volume studio.
[0344] The omnidirectional video output device 221 generates a video signal for re-projecting the omnidirectional video supplied from the background video generating device 53 according to K (K>0) wall displays 222-1 to 222-K. Information about the position, orientation, and size of each of the K wall displays 222-1 to 222-K is given to the omnidirectional video output device 221. The omnidirectional video output device 221 supplies the video signal generated corresponding to the setting of the wall displays 222-1 to 222-K to the wall displays 222-1 to 222-K.
[0345] Each of the wall displays 222-1 to 222-K displays (a portion of) an omnidirectional video based on a video signal from the omnidirectional video output device 221. A synchronization signal generated by the omnidirectional video output device 221 is input to the wall displays 222-1 to 222-K, and the K wall displays 222-1 to 222-K display the omnidirectional video synchronously.
[0346] Note that in the generation of the 3D model data of the person ACT2 as the performer performed by the volumetric video generation device 72, the omnidirectional videos displayed on the wall displays 222-1 to 222-K are cancelled as appropriate.
[0347] <Omnidirectional video output processing>
[0348] Will refer to Fig.33 The flowchart of describes the omnidirectional video output process of displaying the omnidirectional video on K wall displays 222 by the omnidirectional video output device 221 in the image processing system 1 of the fifth embodiment. For example, this process starts when the omnidirectional video is input from the background video generating device 53 to the omnidirectional video output device 221.
[0349] First, in step S151 , the omnidirectional video output device 221 substitutes 1 into a variable k for identifying K wall displays 222 .
[0350] In step S152, the omnidirectional video output device 221 performs a re-projection process of the omnidirectional video, so as to re-project the omnidirectional video on the kth wall display 222 (wall display 222-k). Fig.34 The flowchart describes the details of the re-projection process of omnidirectional video.
[0351] In step S153, the omnidirectional video output device 221 determines whether the variable k is equal to the number K of the wall displays 222. In the case where it is determined in step S153 that the variable k is not equal to the number K of the wall displays 222, in other words, the variable k is less than the number K, the process proceeds to step S154, and the variable k is incremented by 1. Then, the process returns to step S152, and the processes of the above-mentioned steps S152 and S153 are performed on the next wall display 222.
[0352] Meanwhile, in the case where it is determined in step S153 that the variable k is equal to the number K of the wall displays 222, Fig.33 The omnidirectional video output processing is completed.
[0353] Fig.34 is shown as Fig.33 Flowchart of details of the re-projection process of the omnidirectional video performed in step S152.
[0354] First, the omnidirectional video output device 221 sets the y coordinate of the pixel of interest (x, y) for determining the output video of the k-th wall display 222 to 1 in step S171 , and sets the x coordinate to 1 in step S172 .
[0355] In step S173 , the omnidirectional video output device 221 calculates RGB values based on the omnidirectional video and information on the position, orientation and size of the kth wall display 222 , where the RGB values are color information to be displayed on the pixel of interest (x, y) of the kth wall display 222 .
[0356] In step S174 , the omnidirectional video output device 221 writes the RGB value obtained by calculation as the pixel value of the pixel of interest (x, y) of the kth wall display 222 .
[0357] In step S175 , the omnidirectional video output device 221 determines whether the value of the x coordinate of the current pixel of interest (x, y) is the same as the width width of the video size of the output video.
[0358] In the case where it is determined in step S175 that the value of the x coordinate of the current pixel of interest (x, y) is not the same as the width width of the video size of the output video, the process proceeds to step S176, and the value of the x coordinate is incremented by 1. Thereafter, the process returns to step S173, and the processes of steps S173 to S175 described above are repeated. That is, the process of calculating and writing the RGB value as the color information to be displayed is performed using another pixel in the same row of the output video as the pixel of interest (x, y).
[0359] At the same time, when it is determined in step S175 that the value of the x-coordinate of the current pixel of interest (x, y) is the same as the width width of the video size of the output video, the processing proceeds to step S177, and the omnidirectional video output device 221 determines whether the value of the y-coordinate of the current pixel of interest (x, y) is the same as the height height of the video size of the output video.
[0360] In the case where it is determined in step S177 that the value of the y coordinate of the current pixel of interest (x, y) is not the same as the height height of the video size of the output video, the process proceeds to step S178, and the value of the y coordinate is incremented by 1. Thereafter, the process returns to step S172, and the processes of the above steps S172 to S177 are repeated. That is, the processes of the above steps S172 to S177 are repeated until all the lines of the output video are set as the pixel of interest (x, y).
[0361] When it is determined in step S177 that the value of the y coordinate of the current pixel of interest (x, y) is the same as the height height of the video size of the output video, the processing proceeds to step S179, and the omnidirectional video output device 221 outputs the video signal of the output video in which the RGB values of all pixels are written to the kth wall display 222.
[0362] As mentioned above, as Fig.33 The re-projection process of the omnidirectional video performed in step S152 is completed, and the process proceeds to Fig.33 Step S153.
[0363] According to the fifth embodiment of the above-mentioned image processing system 1, a surrounding video imaged by an omnidirectional camera device 58 separated from a real camera device 51 in a background imaging studio or a location scene can be displayed around a performer in a volumetric studio. Therefore, the performer in the volumetric studio can perform while feeling the realism of the background imaging studio or the location scene.
[0364] <13. Summary of the Image Processing System of the Present Disclosure>
[0365] According to the image processing system 1 according to the first to fifth embodiments described above, the background imaging system 11 generates a background video (RGB-D) imaged in a background imaging studio, a location site, or the like, and supplies the background video to the video synthesizing device 31. In addition, the volumetric 2D video generating device 22 generates a volumetric 2D video (RGB-D) of a 3D model of a person in the volumetric studio viewed from a predetermined virtual viewpoint (viewpoint of a virtual camera), and supplies the generated video to the video synthesizing device 31. As the viewpoint of the virtual camera at this time, the volumetric 2D video generating device 22 generates the volumetric 2D video (RGB-D) by using the viewpoint of the real camera 51 (stereo camera 54) of the background imaging system 11. The video synthesizing device 31 generates a synthesized video (RGB) by synthesizing the background video (RGB-D) generated by the background imaging system 11 with the volumetric 2D video (RGB-D) generated by the volumetric 2D video generating device 22. The composite video (RGB) is generated based on the depth information of each of the background video (RGB-D) and the volumetric 2D video (RGB-D), with priority given to the subject on the closer side. The 2D video transmission device 32 sends (transmits) the composite video (RGB) as a transmission video to the client device 33 of the viewing client.
[0366] The background video (RGB) imaged by the real camera 51 is used as the background video of the person who is a performer in the volumetric studio, and thus, a 2D video can be generated as if the person in the volumetric studio is in a background imaging studio, location site, etc. where the real camera 51 exists, and the 2D video is sent to the client device 33.
[0367] Since there is no need to create a 3D CG image for the background by using the background video imaged by the real camera 51, the image can be transmitted immediately at a low cost. The imaging of the background video only requires the acquisition function of the real camera 51 and the position, orientation, zoom value, etc. of the real camera 51, and no advanced imaging equipment is required, and the background video (RGB-D) can be imaged at any place. Therefore, it is easy to create a transmission video, as if the performer actually appears in various imaging places. Since the position, orientation, zoom value, etc. of the real camera 51 are acquired as virtual viewpoint information, and a volume 2D video (RGB-D) is generated, when the real camera 51 moves, zooms, etc., the changes in the background video and the performer's video are accurately matched, and therefore, it is possible to allow the viewing customer to feel natural without giving the feeling of a synthetic video. The imaging place of the background video is not limited to the studio, and can be an outdoor place, such as a place where an accident occurs or a scene of a sports or event. Using the live video of the imaging position, a sense of the scene can be generated and real-time imaging video can be utilized. 2D transmission that can generate a sense of the scene and a sense of reality can be achieved at a low cost.
[0368] Note that in each of the above-mentioned embodiments, the image processing system 1 is formed so that the background imaging system 11 can not only obtain a 2D video of the background, but also obtain depth information, can compare the distance to the subject with the volumetric 2D video (RGB-D), and can generate a composite video (RGB).
[0369] However, in an imaging location or the like where the 2D video imaged by the background imaging system 11 is continuously used as a background without fail, the background imaging system 11 may generate only a 2D video used as a background and omit the output of depth information. In this case, the video synthesizing device 31 generates a synthesized video (RGB) by synthesizing the volumetric 2D video (RGB) generated by the volumetric 2D video generating device 22 and the foreground and background videos (RGB) generated by the background imaging system 11 as a background.
[0370] <About the system configuration of the image processing system>
[0371] In the image processing system 1 , the background imaging system 11 is installed at a first place such as a background imaging studio or a location site, and the volume imaging system 21 and the volume 2D video generating device 22 are installed in a volume studio which is a second place different from the first place.
[0372] On the other hand, the video synthesis device 31 and the 2D video transmission device 32 of the image processing system 1 can be installed in any place, and as in the first embodiment, the video synthesis device 31 can be installed in a video synthesis center, and the 2D video transmission device 32 can be installed in a transmission center. Alternatively, the video synthesis device 31 and the 2D video transmission device 32 can be installed in the same background imaging studio or location site as the background imaging system 11, or can be installed in the same volume studio as the volume imaging system 21 and the volume 2D video generation device 22.
[0373] In the case where the video synthesizing device 31 and the 2D video transmitting device 32 are installed at the same place as the background imaging system 11, one image processing device may be provided, which includes a background video generating unit, a video synthesizing unit, and a 2D video transmitting unit as functions of the background video generating device 53, the video synthesizing device 31, and the 2D video transmitting device 32. Alternatively, the background video generating device 53 and the video synthesizing device 31 may be formed as one image processing device.
[0374] In the case where the video synthesizing device 31 and the 2D video transmitting device 32 are installed at the same place as the volumetric imaging system 21, one image processing device may be provided, which includes a volumetric 2D video generating unit, a video synthesizing unit, and a 2D video transmitting unit as functions of the volumetric 2D video generating device 22, the video synthesizing device 31, and the 2D video transmitting device 32. Alternatively, the volumetric 2D video generating device 22 and the video synthesizing device 31 may be configured by one image processing device.
[0375] <14. Computer Configuration Example>
[0376] The above series of processes can be performed by hardware or software. In the case where a series of processes are performed by software, a program configuring the software is installed in a computer. Here, examples of computers include, for example, a microcomputer built into dedicated hardware, a general-purpose personal computer that can perform various functions by installing various programs, and the like.
[0377] Fig.35 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.
[0378] In the computer, a central processing unit (CPU) 401 , a read only memory (ROM) 402 , and a random access memory (RAM) 403 are connected to each other via a bus 404 .
[0379] The input and output interface 405 is also connected to the bus 404. An input unit 406, an output unit 407, a storage unit 408, a communication unit 409, and a drive 410 are connected to the input and output interface 405.
[0380] The input unit 406 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 407 includes a display, a speaker, an output terminal, etc. The storage unit 408 includes a hard disk, a RAM disk, a nonvolatile memory, etc. The communication unit 409 includes a network interface, etc. The drive 410 drives a removable recording medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.
[0381] In the computer having the above configuration, for example, the CPU 401 loads the program stored in the storage unit 408 into the RAM 403 via the input and output interface 405 and the bus 404, and executes the program, so that the above-mentioned series of processing is performed. In addition, the RAM 403 also appropriately stores data and the like required for the CPU 401 to perform various types of processing.
[0382] For example, the program executed by the computer (CPU 401) can be provided by being recorded on the removable recording medium 411 as a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.
[0383] In the computer, a removable recording medium 411 is mounted in a drive 410 so that the program can be installed into a storage unit 408 via an input and output interface 405. Furthermore, the program can be received by a communication unit 409 via a wired or wireless transmission medium and installed in the storage unit 408. In addition, the program can be pre-installed in the ROM 402 or the storage unit 408.
[0384] Note that, meanwhile, the program executed by the computer may be a program whose processes are executed time-sequentially in the order described in this specification, or may be a program whose processes are executed in parallel or at a required timing such as when a call is issued.
[0385] In this specification, it goes without saying that the steps described in the flowchart can not only be executed chronologically in the described order, but also can be executed in parallel or executed as needed at the timing when calling, etc. even if not processed chronologically.
[0386] In this description, a system means a group of multiple components (devices, modules (parts), etc.), and it is not important whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected to each other via a network and a device in which multiple modules are housed in one housing are both systems.
[0387] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the technical gist of the present disclosure.
[0388] For example, a mode obtained by combining all or some of the above-described plural embodiments may be adopted.
[0389] For example, the technology according to the present disclosure can provide a configuration of cloud computing in which one function is shared and processed by a plurality of devices cooperating with each other via a network.
[0390] Furthermore, each step described in the above flowchart may be performed by one device, or may be shared and performed by a plurality of devices.
[0391] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device or may be executed by a plurality of devices in a shared manner.
[0392] Note that the effects described in this specification are merely examples and not limitations, and there may be effects other than the effects described in this specification.
[0393] Note that the technology of the present disclosure may have the following configurations. (1)
[0395] An image processing device includes: a 2D video generation unit, which obtains camera position information of a camera that performs imaging at a first place as virtual viewpoint information, and generates a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second place different from the first place from the viewpoint of the camera. (2)
[0397] In the image processing apparatus according to (1) above, the 2D video generation unit generates a 2D video and a depth video of the person viewed from a viewpoint of the camera. (3)
[0399] In the image processing device according to any one of (1) or (2) above, a frame number is included in the virtual viewpoint information, and the 2D video generation unit assigns the frame number to the generated 2D video and outputs the 2D video. (4)
[0401] The image processing apparatus according to any one of (1) to (3) above further includes a video synthesis unit that generates a synthesized image obtained by synthesizing the 2D video generated by the 2D video generation unit and the 2D video generated by the camera. (5)
[0403] In the image processing apparatus according to (4) above, the video synthesis unit selects a closer subject from the 2D video generated by the 2D video generation unit and the 2D video generated by the camera, and generates the synthesized image. (6)
[0405] In the image processing apparatus according to (4) or (5) above, the video synthesis unit generates the synthesized image by synthesizing the 2D video generated by the 2D video generation unit and the 2D video generated by the camera device having the same frame number. (7)
[0407] The image processing device according to any one of (1) to (6) above includes a plurality of the 2D video generating units, and the plurality of the 2D video generating units acquire camera position information from different cameras as the virtual viewpoint information and generate 2D videos. (8)
[0409] The image processing device according to any one of (1) to (6) above includes a plurality of the 2D video generating units, and the plurality of the 2D video generating units obtain camera position information from the same camera as the virtual viewpoint information and generate 2D videos. (9)
[0411] The image processing device according to any one of (1) to (8) above further includes: a plurality of video synthesis units that synthesize the 2D videos generated by the plurality of the 2D video generation units and the 2D videos imaged by the plurality of the camera devices. (10)
[0413] The image processing device according to (9) above further includes: a selection unit that selects any one of the plurality of synthesized images generated by the plurality of video synthesizing units and outputs the selected synthesized image. (11)
[0415] In the image processing apparatus according to any one of (1) to (10) above, the camera is a camera that outputs a 2D video and a depth video. (12)
[0417] In the image processing device according to any one of (1) to (11) above, the imaging system including the camera device includes a mode selection unit, which switches between a mode in which the origin position moves in conjunction with the movement of the camera device, a mode in which the origin position is fixed, and a mode in which the origin position can be corrected. (13)
[0419] In the image processing apparatus according to any one of (1) to (12) above, the camera is a camera of a smartphone. (14)
[0421] In the image processing apparatus according to any one of (1) to (12) above, the imaging device is an imaging device of a drone. (15)
[0423] In the image processing apparatus according to any one of (1) to (14) above, the 2D video generation unit acquires lighting information of the first place, and outputs the acquired lighting information to a lighting control device that controls a lighting device at the second place. (16)
[0425] In the image processing device according to any one of (1) to (15) above, the imaging system including the camera device includes a second camera device for imaging the periphery of the camera device, and the video of the second camera device is configured to be displayed on a display at the second location. (17)
[0427] In the image processing device according to any one of (1) to (16) above, the 2D video generation unit acquires the virtual viewpoint information by using a FreeD protocol. (18)
[0429] An image processing system includes: a 2D video generating device that acquires camera device position information of a camera device that performs imaging at a first place as virtual viewpoint information, and generates a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second place different from the first place from the viewpoint of the camera device; and a video synthesizing device that generates a synthesized image obtained by synthesizing the 2D video generated by the 2D video generating device and the 2D video generated by the camera device.
[0430] Reference numerals list
[0431] 1 Image Processing System
[0432] 11 Background Imaging System
[0433] 21 Volume Imaging System
[0434] 22 Volumetric 2D Video Generation Device
[0435] 31 Video synthesis device
[0436] 32 2D video transmission device
[0437] 33 Client Devices
[0438] 51D camera device
[0439] 51R camera device
[0440] 52 Camera motion detection sensor
[0441] 53 Background video generation device
[0442] 54 Stereoscopic camera
[0443] 55 Mode selection button
[0444] 56 Origin position specification button
[0445] 57 Lighting Sensor
[0446] 58 Omnidirectional camera
[0447] 71-1 to 71-N Camera device
[0448] 72 Volumetric Video Generator
[0449] 73 Virtual Camera Device
[0450] 81 Switcher
[0451] 82 Synthetic video selection device
[0452] 141 Smartphone
[0453] 142 Camera
[0454] 151 Drone
[0455] 152 Camera
[0456] 181 Lighting control device
[0457] 182 lighting fixtures
[0458] 201 Illuminance sensor
[0459] 221 Omnidirectional video output device
[0460] 222-1 to 222-k Wall Displays
[0461] 402 ROM
[0462] 404 Bus
[0463] 405 Input and Output Interfaces
[0464] 406 Input Unit
[0465] 407 Output Unit
[0466] 408 Storage Unit
[0467] 409 Communication Unit
[0468] 410 Driver
[0469] 411 Removable recording media
[0470] ACT1 to ACT3 people
Claims
1. An image processing device, comprising: A 2D video generating unit that acquires camera position information of a camera that performs imaging at a first place as virtual viewpoint information, and generates a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second place different from the first place from the viewpoint of the camera.
2. The image processing apparatus according to claim 1, wherein: The 2D video generation unit generates a 2D video and a depth video of the person viewed from a viewpoint of the camera.
3. The image processing apparatus according to claim 1, wherein: The frame number is included in the virtual viewpoint information, and The 2D video generation unit assigns the frame number to the generated 2D video and outputs the 2D video.
4. The image processing device according to claim 1, further comprising: A video synthesis unit generates a synthesized image obtained by synthesizing the 2D video generated by the 2D video generation unit and the 2D video generated by the camera.
5. The image processing apparatus according to claim 4, wherein: The video synthesis unit selects a closer subject from the 2D video generated by the 2D video generation unit and the 2D video generated by the camera, and generates the synthesized image.
6. The image processing apparatus according to claim 4, wherein: The video synthesis unit generates the synthesized image by synthesizing the 2D video generated by the 2D video generation unit and the 2D video generated by the camera having the same frame number.
7. The image processing device according to claim 1, comprising: A plurality of said 2D video generating units, The plurality of 2D video generation units obtain camera position information from different camera devices as the virtual viewpoint information, and generate 2D videos.
8. The image processing device according to claim 1, comprising: A plurality of said 2D video generating units, The plurality of 2D video generating units obtain camera position information from the same camera as the virtual viewpoint information, and generate 2D videos.
9. The image processing device according to claim 1, further comprising: A plurality of video synthesis units synthesize the 2D videos generated by the plurality of the 2D video generation units and the 2D videos imaged by the plurality of the camera devices.
10. The image processing device according to claim 9, further comprising: A selection unit selects any one of the plurality of synthesized images generated by the plurality of video synthesizing units and outputs the selected synthesized image.
11. The image processing apparatus according to claim 1, wherein: The camera device is a camera device that outputs 2D video and depth video.
12. The image processing apparatus according to claim 1, wherein: An imaging system including the camera includes a mode selection unit that switches between a mode in which an origin position moves in conjunction with movement of the camera, a mode in which the origin position is fixed, and a mode in which the origin position can be corrected.
13. The image processing apparatus according to claim 1, wherein: The camera device includes a camera device of a smart phone.
14. The image processing apparatus according to claim 1, wherein: The camera device includes a camera device of a drone.
15. The image processing apparatus according to claim 1, wherein: The 2D video generation unit acquires lighting information of the first place, and outputs the acquired lighting information to a lighting control device that controls a lighting device at the second place.
16. The image processing apparatus according to claim 1, wherein: The imaging system including the camera includes a second camera that images the periphery of the camera, and a video of the second camera is configured to be displayed on a display at the second location.
17. The image processing apparatus according to claim 1, wherein: The 2D video generating unit obtains the virtual viewpoint information by using the FreeD protocol.
18. An image processing system, comprising: a 2D video generating device that acquires camera position information of a camera that performs imaging at a first place as virtual viewpoint information, and generates a 2D video obtained by viewing a 3D model of a person generated by performing imaging at a second place different from the first place from the viewpoint of the camera; and A video synthesis device generates a synthesized image obtained by synthesizing the 2D video generated by the 2D video generation device and the 2D video generated by the camera device.
