Video terminal, video viewing method, and program

Abstract

To enable use of real objects in real space while wearing an HMD.SOLUTION: A student terminal 30 for viewing a lecture held in an immersive virtual space is provided, the student terminal comprising a VR function unit 301 for displaying a virtual space based on virtual space information, and an input unit 302 for receiving a captured video image of real space within arm's reach of a student viewing the lecture. The VR function unit 301 extracts an area with a desk top plate corresponding to a desk object in the virtual space from the video image and inserts a video image within the area to a top plate portion of the desk object for synthesis.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a technique for viewing classes conducted in a virtual space.

Background Art

[0002] In recent years, not only communication education using television and radio but also communication education systems using information and communication technology (ICT) have spread.

[0003] As one of the communication education systems using ICT, it is conceivable to use immersive virtual reality (VR) technology, for example, where students wear a head-mounted display (HMD) and conduct classes in a virtual space. By participating in classes in an immersive virtual space, external information can be shut down, and students can concentrate more on the classes compared to when using a television or a personal computer.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in an immersive virtual space where the field of view is covered by an HMD or the like, physical objects such as textbooks, notebooks, and personal computers on the desk in the real space cannot be used without removing the HMD. The operation of putting on and taking off the HMD during classes conducted in the virtual space is troublesome.

[0006] When it is detected that a user wearing an HMD is looking downward, a technique is known in which an image captured by a camera provided in the HMD is displayed in a virtual space. Using this technique, it is possible to use textbooks and notes in the real space without removing the HMD. However, the conventional method does not consider the objects arranged in the virtual space and displays the image of the real space at the lower part of the virtual space, and there is a problem that the immersion feeling is hindered.

[0007] Patent Document 1 discloses a technique of synthesizing a virtual image with a character drawn thereon and a real image, and causing the character to perform a realistic motion according to the figure included in the real image. The technique of Patent Document 1 synthesizes a virtual character in the real space, and does not synthesize the image of the real object in the real space in the virtual space.

[0008] Patent Document 2 discloses a technique of inputting handwritten characters in a virtual space. The technique of Patent Document 2 displays an object corresponding to an operation panel (acrylic plate) in the real space in the virtual space. When the user writes characters on the operation panel with a finger in the real space, the written characters are recognized and displayed on the object in the virtual space. The user can input handwritten characters into the virtual space using the operation panel in the real space, but the characters written on the operation panel in the real space are not left, and cannot be used for creating notes in the real space.

[0009] The present invention has been made in view of the above, and an object thereof is to make it possible to use real objects in the real space while wearing an HMD.

Means for Solving the Problems

[0010] A viewing terminal according to an aspect of the present invention is a viewing terminal for viewing content in a virtual space, including a head-mounted display for displaying the virtual space based on virtual space information, and inputting control information for controlling an avatar in the virtual space Control information input unit and a camera included in the head-mounted display, and inputting a real space image captured by the cameraVideo An input unit and a control unit that controls the avatar based on the movement of the operator's head detected by the head-mounted display and the control information. The operator selects the plane detected in the real space, The avatar is arranged in the virtual space. corresponding to the selected plane, The real-space video captured by the camera is fitted and synthesized into the virtual space visible from the perspective of the avatar.

Advantages of the Invention

[0011] According to the present invention, it is possible to make use of real objects in the real space while wearing the HMD.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0013] [System Configuration] Hereinafter, a teaching content viewing system according to an embodiment of the present invention will be described with reference to the drawings.

[0014] The viewing system for teaching content in this embodiment is a system that distributes teaching content using virtual reality (VR). The viewing system for teaching content shown in FIG. 1 includes a communication teaching server 10 and a student terminal 30. The communication teaching server 10 and the student terminal 30 are connected via a communication line 20. The communication line 20 is an arbitrary communication line such as the Internet.

[0015] In FIG. 1, only one student terminal 30 is shown, but the number of student terminals 30 is arbitrary. One student terminal 30 is prepared for each student. When the teaching content is distributed as a live broadcast, a large number of student terminals 30 are simultaneously connected to the communication teaching server 10. When the teaching content is distributed on demand, the student terminal 30 and the communication teaching server 10 may be connected one-on-one.

[0016] The student terminal 30 can use any computer device such as a smartphone and a personal computer as long as it has a VR function. An HMD 31 and a camera 32 are connected to the student terminal 30. The camera 32 may be provided in the HMD 31 or may be fixed in the room where the student is located. A device integrating the student terminal 30, the HMD 31, and the camera 32 may also be used.

[0017] When viewing the teaching content, the student wears the HMD 31. The student terminal 30 receives information for constructing a virtual space from the communication teaching server 10, constructs a virtual space based on the received information, and displays an image of the virtual space captured by the HMD 31. When rendering the virtual space, the student terminal 30 fits and synthesizes the image of the student's desk captured by the camera 32 into the desk object in the virtual space.

[0018] To the communication class server 10, a classroom display 11, a relay camera 12, a teacher's terminal 13, a microphone 14, and a speaker 15 are connected. The speaker 15 includes headphones (earphones). The communication class server 10 creates class content within a virtual space based on the information received from the connected devices.

[0019] As shown in FIG. 2, the teacher 40 stands in front of the classroom display 11 and conducts a class using the classroom display 11 and the teacher's terminal 13. The classroom display 11 is a large-screen display called an electronic blackboard. The teacher 40 conducts a class by displaying the class data stored in the communication class server 10 on the classroom display 11. The class data may also be displayed on the teacher's terminal 13. The class data transitions and is displayed in accordance with the speech of the teacher 40, and the class proceeds. The teacher 40 may operate the teacher's terminal 13 to select the class data to be displayed on the classroom display 11.

[0020] The class data displayed on the classroom display 11 is also displayed on a blackboard object arranged within the virtual space. Instead of the classroom display 11, a blackboard or a whiteboard may be used.

[0021] The relay camera 12 photographs the class by the teacher 40. The video data of the teacher 40 being photographed is transmitted to the communication class server 10. The communication class server 10 detects the movement and posture of the teacher 40 from the video data and generates motion data for reflecting it on a teacher avatar arranged within the virtual space. The teacher avatar is a computer graphics (CG) character that reflects the movement of the teacher 40. For example, the communication class server 10 detects the bones (skeleton) of the teacher 40 from the video data and moves the teacher avatar based on the detected bones. The communication class server 10 distributes the motion data of the teacher avatar generated based on the bones to the student terminal 30. Instead of creating the motion data of the teacher avatar from the video data of the teacher 40 being photographed, the teacher may wear an HMD and operate a controller to operate the teacher avatar.

[0022] The microphone 14 picks up the voice emitted by the teacher 40. The picked-up voice data is transmitted to the communication class server 10. The communication class server 10 distributes the voice data to the student terminals 30.

[0023] The speaker 15 outputs, for example, instructions regarding the progress of the class from the filming staff and conveys them to the teacher 40. When the class is two-way, the speaker 15 may output the voice data transmitted from the student terminals 30.

[0024] In the case of video for filming the class, any video may be used to create class content. For example, the bones of the characters may be detected from the video of an educational program recorded in the past, and the avatar in the virtual space may be moved based on the detected bones.

[0025] [Server Configuration] Referring to FIG. 3, the configuration of the communication class server 10 will be described. The communication class server 10 shown in FIG. 3 includes an input / output unit 101, a control unit 102, an audio processing unit 103, a storage unit 104, and a distribution unit 105.

[0026] The input / output unit 101 is connected to the classroom display 11, the relay camera 12, the teacher terminal 13, the microphone 14, and the speaker 15, and outputs class data or inputs video data, audio data, and operations.

[0027] The control unit 102 detects the movements and postures of the teacher 40 from the video data and generates motion data for the teacher avatar. The motion data is stored in the storage unit 104. The control unit 102 stores the information displayed on the classroom display 11 and the information input from the teacher terminal 13 in the storage unit 104.

[0028] The audio processing unit 103 stores the audio received by the microphone 14 and the like in the storage unit 104.

[0029] The storage unit 104 stores information for constructing a virtual space in which a class is held. Examples of information for constructing a virtual space include model data of objects arranged in the virtual space, object arrangement information, model data of an avatar, and motion data. Examples of objects arranged in the virtual space include a blackboard object, objects of the walls, floor, and ceiling of a classroom, and a desk object for students. The storage unit 104 stores class data including voice data recorded according to the progress of the class, information displayed on the classroom display 11, and information input from the teacher terminal 13.

[0030] The distribution unit 105 distributes virtual space information for constructing a virtual space in which a class is held. For example, before the start of a class, the distribution unit 105 distributes model data of objects and avatars arranged in the classroom in the virtual space to the student terminal 30. During the viewing of the class, the distribution unit 105 distributes data that changes according to the progress of the class, such as motion data of the teacher avatar, voice data, and information drawn on the blackboard object, to the student terminal 30 as needed.

[0031] [Terminal Configuration] Referring to FIG. 4, the configuration of the student terminal 30 will be described. The student terminal 30 shown in FIG. 4 includes a VR function unit 301, an input unit 302, a measurement unit 303, an adjustment unit 304, and a storage unit 305.

[0032] The VR function unit 301 receives virtual space information for constructing a virtual space from the communication class server 10 and constructs a virtual space. The VR function unit 301 controls a virtual camera based on the movement of the student's head detected by the HMD 31 and renders the virtual space. The video seen by the student is a virtual space seen from the viewpoint of the student avatar in the virtual space. The student avatar is a CG character in the virtual space of the student who views the class.

[0033] When rendering the virtual space, the VR function unit 301 analyzes the video of the real space captured by the camera 32, extracts the area of the desktop part from the video, and fits and synthesizes the video within the extracted area onto the top plate of the desk object in the virtual space. As a result, the video of the real desktop is merged and displayed on the desk object in the virtual space. Since the area other than the top plate of the desk object where the video is fitted and synthesized is the video rendered in the virtual space, the sense of immersion is not impaired.

[0034] By the student avatar, that is, the student looking at the desk object, the student can see the real desktop projected on the top plate of the desk object while wearing the HMD 31. The student can see textbooks, notes, personal computers, etc. on the real desktop in the virtual space. Since the characters written by the student on the real note are projected onto the desk object, the student can create a real note while wearing the HMD 31. The video of the real space captured by the camera 32 is a video capturing the real space including at least the reach of the student's hand. The student can use real objects in the real space while viewing the video of the real space merged into the virtual space. In addition, the area of the reach of the student's hand extracted from the video may be fitted and synthesized into the virtual space other than the desk object.

[0035] In addition, when the student opens the drawer of the desk, the VR function unit 301 may extract the area of the drawer part from the video, set the desk object in the state where the drawer is opened, and fit and synthesize the area of the drawer part onto the desk object. The desk for fitting and synthesizing the video is not limited to the learning desk, and may be furniture such as a side table, a side wagon, or a side chest. The desk object also includes objects corresponding to these furniture.

[0036] The VR function unit 301 may generate motion data of the student avatar in the virtual space and transmit it to the communication class server 10. The student may operate the controller to control the motion of the student avatar.

[0037] The input unit 302 inputs the video captured by the camera 32. The input unit 302 may input the movement of the student's head and the operation content of the controller detected by the HMD 31. A microphone may be connected to the input unit 302 to input the student's voice, and the input voice data may be transmitted to the communication class server 10.

[0038] Before the student views the class content, the measurement unit 303 measures the three-dimensional space information of the real space where the desk used by the student is placed, and detects the top plate of the desk in the real space based on the obtained three-dimensional space information. For example, while moving the camera 32, a video of the real space is captured, and the three-dimensional space information of the shooting location can be measured by the technology of markerless AR using a monocular camera. The measurement unit 303 detects, for example, a plane at a height within a predetermined range and having a predetermined width as the top plate of the desk based on the obtained three-dimensional space information.

[0039] The measurement unit 303 may present several planes that are candidates for the top plate to the student, and the student may select the plane indicating the top plate. Specifically, for example, the student wears the HMD 31 equipped with the camera 32, and the video captured by the camera 32 is displayed on the HMD 31. The student moves the head (camera 32) so that the top plate of the desk is within the frame of the camera 32. The measurement unit 303 measures the three-dimensional space information from the video and detects the plane. When the measurement unit 303 can detect the plane, a frame indicating the detected plane is superimposed and displayed on the video captured by the camera 32. When the frame indicating the plane coincides with the top plate of the desk, the student performs a determination operation such as operating the controller.

[0040] The adjustment unit 304 adjusts the position of the top plate of the desk object in the virtual space based on the positional relationship between the HMD 31 and the top plate of the desk in the real space. The position of the HMD 31 corresponds to the head of the student avatar. The adjustment unit 304 may adjust the size of the top plate of the desk object to the size of the top plate of the desk in the real space. By adjusting the position and size of the desk object by the adjustment unit 304 to match the desk in the real space, when the student touches the top plate of the desk object with the student avatar, the student can also touch the top plate of the desk in the real space. The size of the student avatar is adjusted according to the student. An object that can be touched in the real space is displayed in the virtual space.

[0041] The storage unit 305 stores information necessary for the VR function unit 301 to render the virtual space, three-dimensional space information measured by the measurement unit 303, and information on the adjusted desk object adjusted by the adjustment unit 304. The storage unit 305 may store information input by the input unit 302 such as an image captured by the camera 32.

[0042] In addition, when preparing a dedicated booth for students to view teaching content, the camera 32 may be fixedly arranged so that the desk used by the students is captured. Use a desk of a determined size and arrange it at a determined position. As a result, since the positions of the camera 32 and the desk are fixed, it is not necessary to detect the top plate by the measurement unit 303. Since the size of the desk is also determined, it is not necessary to adjust the desk object by the adjustment unit 304. Also, in the student's own classroom, the camera 32 may be fixedly arranged. If the desk object is adjusted once according to the desk in the student's own classroom, adjustment is not required from the second time onwards.

[0043] For the communication class server 10 and the student terminal 30, for example, a general-purpose computer system including a central processing unit (CPU), a memory, a storage, a communication device, and an input / output device can be used. In this computer system, the communication class server 10 and the student terminal 30 are realized by the CPU executing a predetermined program loaded onto the memory. This program can be recorded on a computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be distributed via a network.

[0044] The communication class server 10 may render a virtual space. For example, the student terminal 30 transmits the movement of the student's head detected by the HMD 31 to the communication class server 10. The communication class server 10 controls a virtual camera based on the movement of the student's head and renders a virtual space. The student terminal 30 receives and displays the rendered video.

[0045] [Fitting composition processing] Referring to FIG. 5, the flow of the process for performing fitting composition will be described.

[0046] In step S11, the measurement unit 303 detects the position and size of the desk in the real space.

[0047] In step S12, the adjustment unit 304 adjusts the position and size of the desk object in the virtual space based on the position and size of the desk in the real space. For example, based on the distance from the HMD 31 to the top plate of the desk, the distance from the viewpoint position of the student avatar to the top plate of the desk object is adjusted, and further, the size of the top plate of the desk object is adjusted.

[0048] After the adjustment unit 304 adjusts the desk object, the student wears the HMD 31 to display the desk object in the virtual space, moves the hand, and checks whether the position and size of the desk object in the virtual space correspond to the position and size of the desk in the real space, and may finely adjust the position and size of the desk object.

[0049] Through the processes of step S11 and step S12 above, the desk object in the virtual space can be made to correspond to the desk in the real space.

[0050] In step S13, the area on the desk is extracted from the video captured by the camera 32, and the video within the extracted area is fitted and synthesized onto the top plate of the desk object in the virtual space. As a result, the student can view the top of the desk in the real space within the virtual space while wearing the HMD31.

[0051] When the student writes on the note on the desk in the real space, the student's hand and arm will be reflected in the video of the desk being photographed. In this case, the hand and arm of the student avatar may be made invisible or transparent.

[0052] When the VR function unit 301 renders an object, it may highlight the part that corresponds to the physical object in the real space. For example, regarding the top plate of the desk that corresponds to the physical object in the real space, the VR function unit 301 highlights the edge of the top plate of the desk object with a frame. As a result, the student can recognize that there is a physical object in the real space in the part surrounded by the frame in the virtual space.

[0053] Even when the student's head swings, ensure that the video of the real space is synthesized without shaking relative to the object.

[0054] The function of fitting and synthesizing the video of the real space may be able to be turned on / off, or the transparency of the video of the real space may be set and then fitted and synthesized. Even when the video of the real space is not fitted and synthesized, the position and size of the desk object in the virtual space are made to correspond to the desk in the real space.

[0055] [Example of fitting and synthesizing] With reference to FIGS. 6 to 8, an example of synthesizing by fitting a video onto a desk object will be described.

[0056] FIG. 6 is a diagram showing an example of a virtual space before video embedding and synthesis. In this figure, a teacher avatar 400, a desk object 500, and a blackboard object 600 are illustrated. Objects other than those illustrated, such as the floor, ceiling, and walls, may be arranged in the virtual space.

[0057] When there are multiple students taking a class or when producing an effect as if multiple students are taking a class, avatars and desk objects of other students may be arranged. The desk objects of other students may be desk objects in which video of the desk top has been embedded and synthesized, or desk objects in which video has not been embedded and synthesized.

[0058] The teacher avatar 400 is a CG character that progresses the class reflecting the movements of teacher 40. The student terminal 30 controls the teacher avatar 400 based on the motion data received from the communication class server 10.

[0059] The desk object 500 shown in FIG. 6 is not adjusted to correspond to the position and size of the desk in the real space. There is a student avatar (not shown) in front of the desk object 500. Video is embedded and synthesized on the top plate 500A of the desk object 500.

[0060] The blackboard object 600 is an object used by teacher 40 according to the progress of the class. It displays the information displayed on the classroom display 11 or the information input from the teacher terminal 13.

[0061] FIG. 7 is a diagram showing an example of a student's desk in the real space. The desk 700 is a desk within the reach of the hand of the student watching the class. The student terminal 30 extracts the area of the top plate 700A of the desk 700 from the video of the desk 700 taken by the camera 32 while the student is watching the class content.

[0062] FIG. 8 is a diagram showing an example of a virtual space in which an image of the desk in FIG. 7 is inserted and synthesized into the virtual space in FIG. 6. In FIG. 8, the image within the area of the top plate 700A in FIG. 7 is inserted and synthesized onto the top plate 500A of the desk object 500 in the virtual space. Also, the position of the top plate 500A is raised slightly, and the size of the top plate 500A is adjusted to match that of the top plate 700A in the real space. The personal computer and notebook placed on the desk in the real space are also projected onto the desk object in the virtual space. A frame surrounding the top plate 500A into which the image of the real space is inserted and synthesized may be displayed.

[0063] In addition to simply inserting and synthesizing the image of the real space into the desk object, for example, the screen of the monitor of a personal computer in the real space may be projected onto an object in the virtual space. Specifically, the keyboard and mouse of the personal computer are placed on the desk so that they can be operated. The monitor of the personal computer is photographed with a camera. A monitor object corresponding to the monitor of the personal computer is arranged in the virtual space. The display area of the monitor is extracted from the image of the photographed real space monitor and inserted and synthesized onto the monitor object in the virtual space. The student can confirm the display shown on the monitor object in the virtual space while operating the personal computer in the real space. Since the monitor object cannot be touched by the student, the position and size in the virtual space may be set arbitrarily.

[0064] Also, the student's own classroom may be photographed with an omnidirectional camera capable of photographing 360-degree images, and the photographed images may be inserted and synthesized onto the walls, floor, and ceiling of the classroom in the virtual space.

[0065] As described above, the viewing system for lesson content of the present embodiment includes a communication lesson server 10 that distributes lessons conducted in a virtual space and a student terminal 30 for viewing lessons. The communication lesson server 10 includes a distribution unit 105 that distributes virtual space information for displaying lessons conducted in the virtual space. The student terminal 30 includes a VR function unit 301 that displays a virtual space based on the virtual space information, and an input unit 302 that inputs an image of the real space within the reach of the hand of the student viewing the lesson. The VR function unit 301 extracts the area in which the top plate 700A of the desk 700 is reflected from the image, and fits and synthesizes the image within the area onto the top plate 500A of the desk object 500. As a result, the student can view the desk on which the student uses without removing the HMD 31, and can also use textbooks, notebooks, personal computers, etc. on the desk.

Description of Reference Numerals

[0066] 10…Communication lesson server 101…Input / output unit 102…Control unit 103…Audio processing unit 104…Storage unit 105…Distribution unit 11…Classroom display 12…Relay camera 13…Teacher terminal 14…Microphone 15…Speaker 20…Communication line 30…Student terminal 301…VR function unit 302…Input unit 303…Measurement unit 304…Adjustment unit 305…Storage unit 31…HMD 32…Camera

Claims

1. A viewing terminal for viewing content in a virtual space, comprising: A head-mounted display for displaying the virtual space based on virtual space information; A control information input unit for inputting control information for controlling an avatar in the virtual space; A camera included in the head-mounted display; A video input unit for inputting real-space video captured by the camera; A control unit for controlling the avatar based on the detected movement of the operator's head by the head-mounted display and the control information; and The operator selects a plane detected in the real space, Places the avatar in the virtual space, and fits and synthesizes the real-space video captured by the camera corresponding to the selected plane into the virtual space visible from the perspective of the avatar. Viewing terminal.

2. A viewing method by a viewing terminal for viewing content in a virtual space, wherein The viewing terminal includes: A head-mounted display for displaying the virtual space based on virtual space information; A control information input unit for inputting control information for controlling an avatar in the virtual space; A camera included in the head-mounted display; and The operator selects a plane detected in the real space, Inputs the real-space video captured by the camera, Places the avatar in the virtual space, controls the avatar based on the detected movement of the operator's head by the head-mounted display and the control information, And fits and synthesizes the real-space video captured by the camera corresponding to the selected plane into the virtual space visible from the perspective of the avatar. Viewing method.

3. A program for operating a computer as a viewing terminal for viewing content in a virtual space, wherein The viewing terminal includes: A head-mounted display for displaying the virtual space based on virtual space information; A control information input unit for inputting control information for controlling an avatar in the virtual space; A camera included in the head-mounted display; and The operator selects a plane detected in the real space, Inputs the real-space video captured by the camera, Places the avatar in the virtual space, controls the avatar based on the detected movement of the operator's head by the head-mounted display and the control information, And fits and synthesizes the real-space video captured by the camera corresponding to the selected plane into the virtual space visible from the perspective of the avatar. Program.

Images