Multi-site space sharing system and synthetic video generation system

Abstract

To generate a single composite video by combining a plurality of site videos captured at a plurality of sites, and to share people and objects between respective sites in the single composite video.SOLUTION: A multi-site space sharing system displays a composite video using a plurality of site videos captured at a plurality of sites. The multi-site space sharing system includes at least: imaging units 10, one or more of which are arranged at each site and acquire respective site videos; a processing unit 20 that performs removal processing of an arbitrary portion for one or more site videos of the plurality of site videos; a composition unit 30 that generates a composite video by using the plurality of site videos including a site video after the removal processing as a part or all of composite materials; and a display unit 40 that displays the composite video.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a multi-site space sharing system and a composite video generation system for displaying a composite video using a plurality of site videos shot at multiple sites to a user.

Background Art

[0002] The applicant has conceived of the inventions shown in the following Patent Documents 1 and 2 as a system for allowing a user to experience an extended virtual space formed by synthesizing real videos around the user on a virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] One of the objects of the present invention is to provide a multi-site space sharing system and a composite video generation system applicable to various uses as a development of the above invention.

Means for Solving the Problems

[0005] In order to solve the above problems, the present invention is a multi-site space sharing system for displaying a composite video using a plurality of site videos shot at multiple sites, which is configured to include at least: a photographing unit arranged at one or more locations at each site to acquire a site video respectively; a processing unit that performs a removal process on an arbitrary part of one or more of the plurality of site videos; a composite unit that generates a composite video using the plurality of site videos including the site video after the removal process as part or all of composite materials; and a display unit that displays the composite video. This configuration generates a single composite video by combining footage from multiple locations, allowing people and objects from each location to be shared within that single composite video. [Effects of the Invention]

[0006] According to the present invention, at least one of the effects described below is achieved. (1) By combining footage from multiple locations, a single composite video can be generated, allowing people and objects from each location to be shared within that single composite video. (2) In recent years, it has become possible to incorporate actual user footage filmed at different locations into a virtual space also known as the metaverse, eliminating the need for communication via CG avatars and enabling more realistic communication between users. (3) Because the user's real-time video can be incorporated into the composite video, it is suitable for applications where there is a high need for users to work collaboratively even when they are in different locations (such as medical procedures, construction work, education, and other production work). [Brief explanation of the drawing]

[0007] [Figure 1] Functional block diagram of the multi-site space sharing system according to the present invention. [Figure 2] A functional block diagram showing an example configuration of the multi-site space sharing system according to the present invention. [Figure 3] A functional block diagram showing an example configuration of the multi-site space sharing system according to the present invention (Example 2). [Figure 4] A functional block diagram showing an example configuration of the multi-site space sharing system according to the present invention (Example 3). [Figure 5] An illustrative diagram of the depth system removal process. [Figure 6] A conceptual diagram of the chroma removal process. [Figure 7] An illustrative diagram showing the combination of depth-based and chromatic removal processing. [Figure 8]Configuration diagram of the multi-site space sharing system according to Example 1. [Figure 9] An illustrative diagram of the composite video in the multi-site spatial sharing system according to Example 1. [Figure 10] Configuration diagram of the multi-site space sharing system according to Example 2. [Figure 11] An illustrative diagram of the composite video in the multi-site spatial sharing system according to Example 2. [Modes for carrying out the invention]

[0008] The details of the present invention will be described below with reference to the drawings.

[0009] <1> Overall structure (Figure 1) The multi-site spatial sharing system according to the present invention is a system for displaying a composite image, which is formed by combining multiple video images taken at each location, to at least one user. The multi-site spatial sharing system shown in Figure 1 comprises at least an imaging unit 10, a processing unit 20, a synthesis unit 30, and a display unit 40, and also includes a measurement unit 50 and a storage unit 60. Each component can be implemented by arbitrarily combining hardware and software, and various configurations can be adopted, such as each component being a separate device, or multiple components being integrated into a single device. Furthermore, in this invention, there are no particular limitations on the number of bases, the number of parts, the location of each part, or the differences in the configuration at each base.

[0010] Next, Figures 2 to 4 show examples of the configuration of the multi-site space sharing system according to the present invention. It goes without saying that in the present invention, any part can be taken from each example of the configuration and combined to create an adapted form. Furthermore, the composite video displayed at each location may be the same video or different videos, and can be selected as appropriate depending on the purpose.

[0011] <1.1>Configuration Example 1 (Figure 2) In this example, a configuration is adopted in which a device equipped with all components is arranged for each base. In the devices installed at each base, the base video from other bases is received, and a composite video is generated as part or all of the composite material together with the base video of the own base, and is displayed on the display unit 40 of the own base, thereby enabling the user to experience the composite video.

[0012] <1.2> Configuration Example 2 (Fig. 3) In this example, in the said Configuration Example 1, the processing unit 20 and the composite unit 30 are borne by a server installed at a location different from each base (such as a data center, etc.). The server receives the base videos from each base, generates a composite video for each base while appropriately performing removal processing, and transmits this composite video to the display unit 40 of each base, thereby enabling the user to experience the composite video.

[0013] <1.3> Configuration Example 3 (Fig. 4) In this example, according to the purpose at each base, a configuration is adopted in which the combination of each part is different for each base. In Fig. 4, at one base (Base 1), only the imaging unit 10 for capturing the people and objects at the base as the base video is arranged, and at the other base (Base 2), the base video from one base (Base 1) is received, a composite video is generated as part or all of the composite material, and is displayed on the display unit 40 of Base 2, thereby enabling the user to experience the composite video.

[0014] Hereinafter, the details of each part and the details of the calculation procedure will be described.

[0015] <2> Imaging Unit (Fig. 1) The imaging unit 10 has a function for acquiring a base video that becomes part or all of the material of the composite video. One or more imaging units 10 are arranged at each of a plurality of bases that acquire the video to be composited among each base.

[0016] <2.1> Types of Base Videos The camera unit 10 can use cameras built into notebook PCs, tablets, and smartphones, webcams and digital cameras equipped with a PC connection interface, fixed cameras permanently installed at the base, built-in cameras of head-mounted displays (hereinafter also referred to as "HMDs") worn by users, or external cameras attached to HMDs. Furthermore, the shooting unit 10 can also use a camera, also known as a depth camera or 3D camera, which is capable of linking depth information to the captured video. Furthermore, in this invention, the field of view of the imaging unit 10 is not particularly limited.

[0017] <3> Measurement unit (Figure 1) The measurement unit 50 is a device equipped with the function of acquiring the user's location information. This "location information" includes at least information that allows for the recognition of the user's posture and position, and the display of the composite image can be changed based on this location information. The measurement unit 50 can use a group of sensors such as an angle sensor, an acceleration sensor, a gyroscope, a motion capture device, a GPS device, a tracker, etc. Furthermore, the measurement unit 50 includes all types, such as those installed at a base station, those built into the HMD worn by the user, those worn separately on the user's body, and those built into the controller used by the user.

[0018] <4> Machining section (Figure 1) The processing unit 20 has the function of removing arbitrary portions from one or more base images obtained by each shooting unit 10, and acquiring the base image after the removal process. This "removal of arbitrary parts" process involves removing a specific portion of the frame images that make up the base video feed and extracting the remaining portion. In this invention, the method for "removing arbitrary parts" is not particularly limited, but typical removal methods include chromatic removal and depth removal. The details of each removal process are described below.

[0019] <4.1> Depth system removal process (Figure 5) Depth-based removal processing is a process that removes data from location video footage based on the depth information values ​​associated with that location video footage. The depth information linked to the base image can be data pre-embedded in the base image when a depth camera is used as the shooting unit 10, data generated separately by image analysis of the base image, or data obtained from a separately installed 3D scanner, etc.

[0020] One example of depth-based image removal is a process that removes portions of a base station's video footage that have depth information exceeding a predetermined distance, based on the depth information of the base station's video footage. For example, if areas with depth information of 50 cm or more are to be removed from the imaging unit 10, the resulting base image will consist only of images extracted from the imaging unit 10 of areas with depth information of less than 50 cm.

[0021] For example, if we assume that the shooting unit 10 is a depth camera that takes pictures of the front of the HMD worn by the user, then, as shown in Figure 5, when depth system removal processing is applied to the base image A1, the resulting base image A2 will be an image in which only the user's hands B are extracted.

[0022] Furthermore, the extraction area using depth-based removal is not limited to the user's body; it can also extract objects such as controllers held by the user and objects surrounding the user (chairs, desks, ladders, food, etc.).

[0023] <4.2> Chroma removal process (Figure 6) Chroma removal is a process that removes unwanted elements from video footage at a given location based on the color space information values ​​of that video footage.

[0024] Chroma removal is performed by removing parts corresponding to specific homogeneous colors by setting a predetermined range within the elements and numerical ranges defined in each color space, based on well-known color space information such as RGB, RGBA, CMY, HSV, and HLS. For example, when performing chroma removal using HSV color space information, the HSV color space information includes parameters such as hue (0°~360°), saturation (0%~100%), and value (0%~100%). By setting the numerical range for each parameter, the parts corresponding to specific homogeneous colors are removed.

[0025] When performing chroma removal processing, it is desirable to install a background material C that exhibits a uniform color in advance in the area included in the field of view of the imaging unit 10 installed at the base.

[0026] <4.3> Combinations of each removal process (Figure 7) In the present invention, the details, order, and number of times the "removal process of arbitrary parts" described above are not particularly limited and can be appropriately determined according to the application of the system according to the present invention. Furthermore, as shown in Figure 7, by performing depth-based decryption on a single location video to be processed, and then applying chroma-based decryption, it is possible to more clearly remove areas that could not be clearly removed by depth-based decryption alone, such as the gaps between the user's fingers and the floor around the user's feet.

[0027] <5> Storage section (Figure 1) The storage unit 60 has the function of storing composite materials other than the base video (hereinafter also referred to as "material video"). The video footage stored in the storage unit 60 can include, in addition to the base location footage, footage of a real location that has been filmed in advance, pre-produced CG footage, virtual footage (VR footage) that provides a VR space, or a combination of these.

[0028] <5.1> Selection of source footage Furthermore, when generating the composite video to be experienced by each user, the same source video may be provided for each composite video, or different source videos may be provided, and this can be appropriately determined depending on the application of the multi-site spatial sharing system according to the present invention. Furthermore, the present invention does not exclude the application of the aforementioned "removal process of arbitrary portions" to the source video.

[0029] <6> Synthesis section (Figure 1) The synthesis unit 30 has the function of generating a composite image by using multiple source images as part or all of the synthesis material. This "multiple location footage" includes both location footage from the shooting unit 10 and location footage after it has been processed by the processing unit 20. Therefore, for all base images, the base images after the removal process may be used as composite material, or some base images may be used as composite material after the removal process, while the remaining base images may be used as composite material without going through the processing unit 20. Furthermore, the composite material may also include the source video footage stored in the aforementioned storage unit 60. The synthesis in the synthesis unit 30 can be done by simply overlapping each synthesis material, or by using positional information obtained from the measurement unit 50, depth information linked to base video, etc., to calculate the position of the material video in the VR space and then overlapping the synthesis material.

[0030] <7> Display unit (Figure 1) The display unit 40 has the function of displaying the composite image generated by the synthesis unit 30 to the user. The display unit 40 is not limited to general displays, monitors, projectors, etc., but can also use head-mounted displays, goggles, or mobile devices such as smartphones that can be housed in a housing to function as goggles, which can be used for spatial experiences such as VR, AR, and XR. [Examples]

[0031] <1> Overall structure (Figure 8) Figure 8 is a diagram illustrating the configuration of the multi-site spatial sharing system described in Configuration Example 1, when used in a medical education setting. Each location is equipped with a head-mounted display Y1 with a first camera X1 which is a depth camera, a second camera X2 which photographs the user U1 (U2) wearing the head-mounted display Y1, and an information processing device Z. Each head-mounted display Y1 has a built-in tracker (not shown) that periodically acquires the position information of the user U1 (U2) wearing the head-mounted display Y1. In this embodiment, the first camera X1 and the second camera X2 at each location function as an imaging unit 10, the information processing device Z functions as at least a processing unit 20, a synthesis unit 30, and a storage unit 60, and the head-mounted display Y1 functions as a display unit 40 and a measurement unit 50.

[0032] In addition, the present invention may also involve assigning the functions of the processing unit 20, the synthesis unit 30, and the storage unit 60 to a server installed in a separately provided data center, as shown in the configuration example 2 described above, instead of the configuration shown in Figure 8 above.

[0033] Next, we will explain the process of generating and displaying the composite video on site 1. In this invention, the following processes can be rearranged or performed in parallel, as long as this does not cause inconsistencies in operation. (1) The base image captured by the first camera X1 is sent to the information processing device Z of base 1, where arbitrary parts are removed (in this example, depth-based removal and chroma-based removal) and only the area around user U1 is extracted to generate a removed base image, which is used as the composite material. (2) Next, the information processing device Z at base 1 receives the base video captured by the second camera X2 at base 2 via the information processing device Z at base 2. In this embodiment, since depth information is not associated with the video captured by the second camera X2, only chroma removal processing is applied to the base video captured by the second camera X2 to generate a base video after the removal process, from which user U2 has been extracted, and this is used as the composite material. (3) Furthermore, the information processing device Z at base 1 stores a virtual image of a patient placed on an operating table as composite material. (4) The information processing device Z of base 1 generates a composite image by appropriately combining the composite materials of (1), (2), and (3). At this time, since depth information is not associated with the base image after the removal process of (2), a virtual display frame is set up in advance on the image of the virtual space of (3), and the base image after the removal process of (2) is composited into this virtual display frame. (5) The composite image generated by the information processing device Z at base 1 is displayed on the head-mounted display Y1 worn by user U1, so as to change the display mode based on the location information of user U1.

[0034] <2> Example of composite image (Figure 9) Figure 9 shows an image of the composite video displayed on each user's head-mounted display Y1 in the multi-site spatial sharing system according to this embodiment.

[0035] <2.1> Composite video from base 1 Figure 9(a) is an image of the composite video as seen by user U1 at site 1. In this composite image, the source video D, which consists of a virtual space displaying an operating table and a patient placed on the operating table as described in (3) above, is composited with the surrounding video of user U1 included in the base video after removal processing as described in (1) above, on the near side of the operating table in the virtual space. Furthermore, the video of user U2 included in the base video after removal processing as described in (2) above is composited into a virtual display frame D1 that is virtually placed on the far side of the operating table. Furthermore, when user U1 moves closer to the operating table, the composite image is also changed to reflect user U1's position closer to the operating table, based on the positional information obtained from user U1's head-mounted display Y1.

[0036] <2.2> Composite video from base 2 Figure 9(b) is an image of the composite video as seen by user U2 at site 2. In this composite image, the post-processed footage of base 1, taken from the second camera X2 at base 1, is superimposed onto the virtual display frame D1, with the operating table in the background of the source footage D. Furthermore, while the surrounding footage of user U2 will be superimposed on the foreground of the operating table in source video D, user U2 has both arms raised, and both arms are not within the field of view of the first camera X1. Therefore, user U2's hands are not displayed in the superimposed footage.

[0037] <3> supplementary explanation In the above explanation, the operating table and patient are provided in a virtual space which is the source video. However, for example, an actual operating table and a model of the patient could be provided at one of the locations. For example, if a physical model is prepared at base 1, the following methods can be adopted. (Method 1) A method in which a first camera X1 on the base site side photographs the actual operating table and a model of the patient, and the distance to be extracted using depth-based removal processing is increased so that the actual operating table and the model of the patient are included in the base site image after removal processing based on the first camera X1. (Method 2) A method in which the video footage of the actual operating table and patient model captured by the second camera X2 on site 1 is not only transmitted to site 2, but is also used as source video for the composite video on site 1.

[0038] <4> summary With the configuration described above, users in remote locations can share a single space while recognizing their own hands, surrounding area, and the other person's appearance in the synthesized image as real-world images. Therefore, it is suitable for applications where there is a high need to mutually confirm each person's actions, such as the medical education described in this embodiment. [Examples]

[0039] <1> Overall structure (Figure 10) Figure 10 is a diagram illustrating the configuration when the multi-site space sharing system described in Configuration Example 3 is used as a service provided by an aquarium. Location 1 is equipped with a display Y2 installed in front of user U3, a third camera X3 mounted above the display Y2 to photograph user U3, and an information processing device Z. Base 2 is equipped with a fourth camera, X4, which films tank T inside the aquarium. The third camera X3 and the fourth camera X4 are both depth cameras capable of linking depth information to the captured video. In this embodiment, the third camera X3 and the fourth camera X4 function as the imaging unit 10, the information processing device Z functions as at least the processing unit 20 and the synthesis unit 30, and the display Y2 functions as the display unit 40.

[0040] The process for generating and displaying the composite video at site 1 is described below. (1) The base station video captured by the third camera X3 is sent to the information processing device Z of base station 1, where arbitrary parts are removed (in this example, depth-based removal and chroma-based removal) and only the area around user U3 is extracted to generate a post-removal base station video, which is used as the composite material. (2) Next, the information processing device Z at base 1 receives the base video footage taken of the aquarium T by the fourth camera X4 at base 2, and generates a degraded base video, which is extracted with depth information linked to the images of the fish and shellfish in the aquarium T, and uses this as the composite material. (3) The information processing device Z at site 1 generates a composite image by appropriately combining the composite materials described in (1) and (2). Since each image contains depth information, the composite can be performed based on a certain depth position.

[0041] <2> Example of composite image (Figure 11) Figure 11 shows an image of the composite video displayed on display Y2 at site 1 in the multi-site spatial sharing system according to this embodiment. The composite image shown in Figure 11 is a composite of footage of the aquarium T with depth information linked to it, as described in (2) above, and footage of user U3 with depth information linked to it. Since both of the base images in (1) and (2) above contain depth information, for example, when a fish moves in front of the tank T, the fish will be displayed as if it is crossing in front of user U3, and when user U3 extends their hand in front of the fish crossing in front of them, the fish will be displayed as if it is hidden by user U3's hand. Therefore, it is possible to display an image on the display Y2 that makes it appear as if user U3 is inside the aquarium T and fish are moving around user U3, allowing user U3 to have the experience of entering the aquarium T and sharing the space with the fish.

[0042] <3> supplementary explanation In the above description, depth information is assumed to be linked to all base video feeds. However, for example, depth information may be linked only to the base video feed of base 2, and the base video feed of base 1 may be composited at a predetermined depth position in the base video feed of base 2.

[0043] <4> summary With the configuration described above, even in situations where it is difficult to open an aquarium, it is possible to provide users in remote locations with opportunities for virtual interaction and to offer experiences that are impossible in reality, such as allowing users to enter the inside of an exhibit tank. Therefore, not only aquariums, but also zoos, amusement parks, sports fields, and other facilities can offer customers new experiences that are different from those available during opening hours. [Examples]

[0044] Furthermore, the present invention can also be used as a composite image generation system comprising at least the processing unit 20 and the synthesis unit 30 described above. The video data used as composite material is not particularly limited; real-time video data or pre-recorded and saved video data can be used as appropriate. Furthermore, the composite image generation system according to this embodiment may also be further equipped with a measurement unit 50 and a storage unit 60. The composite video generation system according to this embodiment can also be applied to video creation work, specifically when creating a composite video by appropriately combining pre-recorded data. [Explanation of symbols]

[0045] 10. Photography Department 20 Processing Department 30 Synthesis section 40 Display section 50 Measurement section 60 Preservation Department A1 Base video A2 Base footage after removal process B Hands C Background material U1, U2 users X1 First Camera X2 Second Camera X3 Third Camera X4 4th Camera Y1 Head-Mounted Display Y2 Display Z Information Processing Device

Claims

1. A multi-site space sharing system that displays a composite image to a user using multiple site images captured at multiple sites, One or more imaging units are provided at each location, and each imaging unit acquires location images associated with depth information; a processing unit that performs a removal process of an arbitrary portion on one or more of the plurality of base images by a depth-based removal process that is a removal process based on the depth information; and a synthesis unit that generates a synthetic image by synthesizing the plurality of site images, including the site image after the removal process, as a part or all of a synthesis material using the respective depth information; A display unit that displays the composite image. Multi-location space sharing system.

2. At one of the locations where the user is present, the display unit is a display not worn by the user, The photographing unit acquires a location image including the user, The composite image displayed on the display is The image is a composite image obtained by combining a site image including the user captured at one site and a site image captured at another site as a composite material in whole or in part using the depth information of each site. The multi-site space sharing system according to claim 1 .

3. A measuring unit that measures the position information of the user, The synthesis unit generates the synthetic image by further using the position information. The multi-site space sharing system according to claim 1 .

4. The method further includes a storage unit for storing a material image that can be used as a part of a composite material by the composite unit. The multi-site space sharing system according to claim 1 .

5. The material image includes a virtual image that provides a VR space. The multi-site space sharing system according to claim 4.

6. A chroma system removal process, the processing unit further being a removal process based on a value of color space information. The present invention is characterized in that The multi-site space sharing system according to claim 1 .

7. A synthetic image generation system, comprising: A processing unit that performs a removal process of an arbitrary part including at least a depth-based removal process, which is a removal process based on the depth information, on two or more pieces of video data associated with depth information among a plurality of pieces of video data captured by different cameras; A synthetic image generation system comprising at least a synthesis unit that generates a single synthetic image data by synthesizing the multiple image data, including the image data after the removal process, using the respective depth information as part or all of a synthetic material.

Images