Viewing terminal, viewing method, and program

Abstract

To use real materials in a real space, with an HMD mounted thereon.SOLUTION: A student terminal 30 for viewing a lesson with a sense of immersion performed in a virtual space, includes a VR function section 301 for displaying a virtual space based on virtual space information, and an input section 302 for inputting a video obtained by capturing an image of a real space within reach of a student viewing the lesson. The VR function section 301 extracts a region where a top plate of a desk corresponding to a desk object in the virtual space is reflected, from the video, and combines it by embedding the video within the range in the top plate part of the desk object.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a technique for viewing a lesson held in a virtual space. [Background technology]

[0002] In recent years, not only distance education using television and radio but also distance education systems using information and communication technology (ICT) have become widespread.

[0003] One of the distance education systems using ICT is to use immersive virtual reality (VR) technology, for example by having students wear a head-mounted display (HMD) and take classes in a virtual space. By participating in classes in an immersive virtual space, students can shut out information from the outside world and concentrate more on the class than when using a television or personal computer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-203786 A [Patent Document 2] JP 2019-101945 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in an immersive virtual space where the field of vision is covered by an HMD or other device, physical objects such as textbooks, notebooks, and computers on a desk in the real world cannot be accessed unless the HMD is removed. The action of putting on and taking off the HMD during classes in a virtual space is troublesome.

[0006] There is a known technology that displays images captured by a camera in the HMD in a virtual space when it detects that the user wearing the HMD is looking downward. Using this technology, you can use textbooks and notebooks in the real world without removing the HMD. However, the conventional method does not take into account objects placed in the virtual space, and displays images of the real world at the bottom of the virtual space, which has the problem of hindering the sense of immersion.

[0007] Patent Document 1 discloses a technology that synthesizes a virtual image of a character with a real image, and makes the character move in a realistic manner according to the figure contained in the real image. The technology in Patent Document 1 synthesizes a virtual character in real space, but does not synthesize an image of an actual object in real space in virtual space.

[0008] Patent Document 2 discloses a technology for inputting handwritten characters in a virtual space. The technology of Patent Document 2 displays an object in the virtual space that corresponds to an operation panel (acrylic panel) in the real space. When a user writes characters on the operation panel with his / her finger in the real space, the written characters are recognized and displayed on an object in the virtual space. The user can input handwritten characters into the virtual space using the operation panel in the real space, but the characters written on the operation panel in the real space are not retained and cannot be used to create notes in the real space.

[0009] The present invention has been made in view of the above, and has an object to enable use of actual objects in real space while wearing an HMD. [Means for solving the problem]

[0010] A viewing terminal according to one aspect of the present invention is a viewing terminal for viewing content in a virtual space, Equipped with a camera, Displaying a virtual space based on virtual space information Head-mounted display a measurement unit for measuring three-dimensional space information of a real space; The camera An input unit for inputting a captured image, beforeextracting a target area from the image based on the three-dimensional space information, and fitting the image within the target area into an object displayed in the virtual space to synthesize the object; placing an avatar of a viewer in the virtual space, and making the avatar's hands or arms invisible or transparent when the user's hands or arms are reflected in the image within the target area; The positional relationship between the viewpoint and the object in the virtual space is The head mounted display and the target area. Effect of the Invention

[0011] According to the present invention, it is possible to use real objects in real space while wearing an HMD. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the overall configuration of a lesson content viewing system according to this embodiment. [Diagram 2] FIG. 2 is a conceptual diagram showing a classroom from which lesson content is distributed. [Diagram 3] FIG. 3 is a functional block diagram showing an example of the configuration of a correspondence class server. [Figure 4] FIG. 4 is a functional block diagram showing an example of the configuration of the student terminal. [Diagram 5] FIG. 5 is a flowchart showing the flow of processing for fitting an image of a desk in real space into a desk object in virtual space. [Figure 6] FIG. 6 is a diagram showing an example of a virtual space before an image is embedded and synthesized. [Figure 7] FIG. 7 is a diagram showing an example of a student's desk in the real space. [Figure 8] FIG. 8 is a diagram showing an example of a virtual space obtained by inserting the image of the desk shown in FIG. 7 into the virtual space shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [System Configuration] Hereinafter, a lesson content viewing system according to one embodiment of the present invention will be described with reference to the drawings.

[0014] The lesson content viewing system of this embodiment is a system that distributes lesson content using virtual reality (VR). The lesson content viewing system shown in Fig. 1 includes a correspondence class server 10 and a student terminal 30. The correspondence class server 10 and the student terminal 30 are connected via a communication line 20. The communication line 20 is any communication line such as the Internet.

[0015] 1 shows only one student terminal 30, the number of student terminals 30 can be any number. One student terminal 30 is provided for each student. When lesson content is distributed by live broadcast, many student terminals 30 are simultaneously connected to the correspondence class server 10. When lesson content is distributed on demand, student terminals 30 and correspondence class server 10 may be connected one-to-one.

[0016] The student terminal 30 may be any computer device such as a smartphone or a personal computer, provided that the device has a VR function. An HMD 31 and a camera 32 are connected to the student terminal 30. The camera 32 may be provided in the HMD 31, or may be fixed in a room where the student is present. A device in which the student terminal 30, the HMD 31, and the camera 32 are integrated may be used.

[0017] When viewing the lesson content, the student wears the HMD 31. The student terminal 30 receives information for constructing a virtual space from the correspondence class server 10, constructs the virtual space based on the received information, and displays an image of the virtual space on the HMD 31. When rendering the virtual space, the student terminal 30 fits an image of the student's desk, captured by the camera 32, into a desk object in the virtual space and synthesizes it.

[0018] A classroom display 11, a relay camera 12, a teacher's terminal 13, a microphone 14, and a speaker 15 are connected to the correspondence class server 10. The speaker 15 includes a headphone (earphone). The correspondence class server 10 creates lesson content in a virtual space based on information received from the connected devices.

[0019] As shown in FIG. 2, teacher 40 stands in front of classroom display 11 and uses classroom display 11 and teacher terminal 13 to teach a class. Classroom display 11 is a large-screen display known as an electronic whiteboard. Teacher 40 teaches by displaying lesson data stored in correspondence class server 10 on classroom display 11. The lesson data may be displayed on teacher terminal 13. The lesson progresses as the lesson data changes and is displayed in accordance with teacher 40's speech. Teacher 40 may operate teacher terminal 13 to select lesson data to be displayed on classroom display 11.

[0020] The lesson data displayed on the classroom display 11 is also displayed on a blackboard object arranged in the virtual space. A blackboard or a whiteboard may be used instead of the classroom display 11.

[0021] The relay camera 12 captures a lesson given by the teacher 40. The video data of the teacher 40 is sent to the correspondence class server 10. The correspondence class server 10 detects the movement and posture of the teacher 40 from the video data and generates motion data to be reflected in a teacher avatar placed in a virtual space. The teacher avatar is a computer graphics (CG) character that reflects the movement of the teacher 40. For example, the correspondence class server 10 detects the bones (skeleton) of the teacher 40 from the video data and moves the teacher avatar based on the detected bones. The correspondence class server 10 distributes the motion data of the teacher avatar generated based on the bones to the student terminal 30. Instead of creating the motion data of the teacher avatar from the video data of the teacher 40, the teacher may wear an HMD and operate a controller to operate the teacher avatar.

[0022] The microphone 14 picks up the voice uttered by the teacher 40. The picked up voice data is transmitted to the correspondence class server 10. The correspondence class server 10 distributes the voice data to the student terminals 30.

[0023] The speaker 15 outputs, for example, instructions from the filming staff regarding the progress of the lesson, and conveys them to the teacher 40. If the lesson is two-way, the speaker 15 may output audio data transmitted from the student terminals 30.

[0024] In addition, any video footage of a lesson may be used to create lesson content. For example, the bones of characters in a previously recorded educational program may be detected, and avatars in a virtual space may be moved based on the detected bones.

[0025] [Server configuration] The configuration of the distance learning server 10 will be described with reference to Fig. 3. The distance learning server 10 shown in Fig. 3 comprises an input / output unit 101, a control unit 102, an audio processing unit 103, a storage unit 104, and a distribution unit 105.

[0026] The input / output unit 101 is connected to the classroom display 11, relay camera 12, teacher terminal 13, microphone 14, and speaker 15, and outputs lesson data and inputs video data, audio data, and operations.

[0027] The control unit 102 detects the movement and posture of the teacher 40 from the video data and generates motion data for the teacher avatar. The motion data is stored in the memory unit 104. The control unit 102 stores in the memory unit 104 the information displayed on the classroom display 11 and the information input from the teacher terminal 13.

[0028] The audio processing unit 103 stores the audio and the like received by the microphone 14 in the storage unit 104 .

[0029] The storage unit 104 stores information for constructing a virtual space in which a lesson is held. Information for constructing a virtual space includes, for example, model data of objects to be placed in the virtual space, object placement information, and avatar model data and motion data. Objects to be placed in the virtual space include, for example, a blackboard object, objects on the walls, floor, and ceiling of the classroom, and student desk objects. The storage unit 104 stores lesson data including audio data recorded as the lesson progresses, information displayed on the classroom display 11, and information input from the teacher's terminal 13.

[0030] The distribution unit 105 distributes virtual space information for constructing a virtual space in which a lesson is held. For example, before the start of a lesson, the distribution unit 105 distributes model data of objects and avatars arranged in a classroom in the virtual space to the student terminals 30. While the lesson is being viewed, the distribution unit 105 distributes data that changes according to the progress of the lesson, such as motion data of the teacher avatar, audio data, and information drawn on a blackboard object, to the student terminals 30 as needed.

[0031] [Device configuration] The configuration of the student terminal 30 will be described with reference to Fig. 4. The student terminal 30 shown in Fig. 4 includes a VR function unit 301, an input unit 302, a measurement unit 303, an adjustment unit 304, and a storage unit 305.

[0032] The VR function unit 301 receives virtual space information for constructing a virtual space from the correspondence class server 10 and constructs the virtual space. The VR function unit 301 controls the virtual camera based on the student's head movement detected by the HMD 31 to render the virtual space. The image seen by the student is the virtual space seen from the viewpoint of the student avatar in the virtual space. The student avatar is a CG character in the virtual space of the student watching the class.

[0033] When rendering the virtual space, the VR function unit 301 analyzes the image of the real space captured by the camera 32, extracts the area of ​​the top of the desk from the image, and synthesizes the image of the extracted area by fitting it onto the top plate of a desk object in the virtual space. This causes the image of the desk top in the real space to be blended with the desk object in the virtual space and displayed. Since the image is a rendered image of the virtual space except for the top plate of the desk object into which the image is fitted and synthesized, the sense of immersion is not lost.

[0034] By the student avatar, that is, the student, looking at the desk object, the student can view the desk in the real space projected onto the top of the desk object while wearing the HMD 31. The student can view textbooks, notebooks, computers, etc. on the desk in the real space in the virtual space. Characters, etc. written by the student in the notebook in the real space are projected onto the desk object, so the student can create a notebook in the real space while wearing the HMD 31. The image of the real space captured by the camera 32 is an image of the real space that includes at least the range within the reach of the student. The student can use the real objects in the real space while viewing the image of the real space merged with the virtual space. Note that the area within the reach of the student extracted from the image may be inserted and synthesized in the virtual space other than the desk object.

[0035] When a student opens a desk drawer, the VR function unit 301 may extract the area of ​​the drawer from within the image, set the desk object to the state where the drawer is open, and fit the area of ​​the drawer into the desk object to synthesize the image. The desk into which the image is fitted and synthesized is not limited to a desk, but may be furniture such as a side table, a side wagon, or a side chest. The desk object also includes objects corresponding to these pieces of furniture.

[0036] The VR function unit 301 may generate motion data of a student avatar in a virtual space and transmit it to the correspondence class server 10. The student may control the motion of the student avatar by operating a controller.

[0037] The input unit 302 inputs the video captured by the camera 32. The input unit 302 may input the head movements of the students and the operation of the controller detected by the HMD 31. A microphone may be connected to the input unit 302 to input the voice of the students, and the input voice data may be transmitted to the distance learning server 10.

[0038] Before the student watches the lesson content, the measurement unit 303 measures three-dimensional spatial information of the real space where the desk used by the student is placed, and detects the desk top in the real space based on the obtained three-dimensional spatial information. For example, a video of the real space is shot while moving the camera 32, and the three-dimensional spatial information of the shooting location can be measured by a markerless AR technique using a monocular camera. For example, the measurement unit 303 detects a flat surface that is within a predetermined height range and has a predetermined width as the desk top based on the obtained three-dimensional spatial information.

[0039] The measuring unit 303 may present several planes that are candidates for the tabletop to the student, and the student may select the plane that represents the tabletop. Specifically, for example, the student wears the HMD 31 equipped with the camera 32, and an image captured by the camera 32 is displayed on the HMD 31. The student moves his / her head (camera 32) so that the tabletop fits within the frame of the camera 32. The measuring unit 303 measures three-dimensional space information from the image to detect the plane. When the measuring unit 303 detects the plane, it displays a frame indicating the detected plane superimposed on the image captured by the camera 32. When the frame indicating the plane matches the tabletop, the student performs a decision operation such as operating a controller.

[0040] The adjustment unit 304 adjusts the position of the top of the desk object in the virtual space based on the positional relationship between the HMD 31 and the desk top in the real space. The position of the HMD 31 corresponds to the head of the student avatar. The adjustment unit 304 may adjust the size of the top of the desk object to the size of the desk top in the real space. By adjusting the position and size of the desk object by the adjustment unit 304 to match the desk in the real space, when a student touches the top of the desk object with the student avatar, the student can also touch the desk top in the real space. The size of the student avatar is adjusted to match the student. Objects that can be touched in the real space are displayed in the virtual space.

[0041] The storage unit 305 holds information necessary for the VR function unit 301 to render the virtual space, three-dimensional space information measured by the measurement unit 303, and information on the desk object after adjustment by the adjustment unit 304. The storage unit 305 may store information input by the input unit 302, such as an image captured by the camera 32.

[0042] When preparing a dedicated booth for students to view lesson content, the camera 32 may be fixedly arranged so as to capture the desk used by the student. A desk of a predetermined size is used and arranged at a predetermined position. Since the positions of the camera 32 and the desk are fixed, there is no need for the measuring unit 303 to detect the table top. Since the size of the desk is also determined, there is no need for the adjusting unit 304 to adjust the desk object. The camera 32 may also be fixedly arranged in the student's own room. Once the desk object is adjusted according to the desk in the student's own room, no adjustment is required from the second time onwards.

[0043] For example, a general-purpose computer system equipped with a central processing unit (CPU), memory, storage, communication devices, and input / output devices can be used for the correspondence class server 10 and the student terminal 30. In this computer system, the CPU executes a predetermined program loaded onto the memory, thereby realizing the correspondence class server 10 and the student terminal 30. This program can be recorded on a computer-readable recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

[0044] The correspondence class server 10 may render the virtual space. For example, the student terminal 30 transmits the head movement of the student detected by the HMD 31 to the correspondence class server 10. The correspondence class server 10 controls the virtual camera based on the head movement of the student and renders the virtual space. The student terminal 30 receives and displays the rendered image.

[0045] [Inlay synthesis processing] The flow of the embedding synthesis process will be described with reference to FIG.

[0046] In step S11, the measurement unit 303 detects the position and size of the desk in the real space.

[0047] In step S12, the adjustment unit 304 adjusts the position and size of the desk object in the virtual space based on the position and size of the desk in the real space. For example, the adjustment unit 304 adjusts the distance from the viewpoint of the student avatar to the top of the desk object based on the distance from the HMD 31 to the top of the desk, and further adjusts the size of the top of the desk object.

[0048] After adjusting the desk object using adjustment 304, the student may wear HMD 31 to display the desk object in the virtual space, and move his / her hands to check whether the position and size of the desk object in the virtual space correspond to the position and size of the desk in the real space, and fine-tune the position and size of the desk object.

[0049] By the above processes in steps S11 and S12, the desk object in the virtual space can be made to correspond to the desk in the real space.

[0050] In step S13, the area of ​​the desk is extracted from the image captured by the camera 32, and the image of the extracted area is fitted onto the top plate of the desk object in the virtual space and synthesized. This allows the student to see the desk in the real space within the virtual space while wearing the HMD 31.

[0051] When a student writes in a notebook on a desk in real space, the student's hands and arms are captured in the image of the desk. In this case, the student's hands and arms may be hidden or made transparent.

[0052] When rendering an object, the VR function unit 301 may highlight a portion that corresponds to an actual object in real space. For example, for a desk top that corresponds to an actual object in real space, the VR function unit 301 highlights the edge of the top of the desk object by surrounding it with a frame. This allows the student to recognize that the portion surrounded by the frame in the virtual space is an actual object in the real space.

[0053] To synthesize an image of real space onto an object without shaking even if a student's head moves.

[0054] The function of embedding and compositing the image of the real space may be turned on / off, or the transparency of the image of the real space may be set and embedded. Even if the image of the real space is not embedded and composited, the position and size of the desk object in the virtual space should correspond to the desk in the real space.

[0055] [Example of embedding] An example of compositing in which an image is embedded in a desk object will be described with reference to FIGS.

[0056] 6 is a diagram showing an example of a virtual space before the video is embedded and synthesized. In the figure, a teacher avatar 400, a desk object 500, and a blackboard object 600 are shown. Objects other than those shown in the figure, such as a floor, ceiling, and walls, may be placed in the virtual space.

[0057] When multiple students are taking a class together or when you want to present the class as if multiple students are taking a class together, you may place avatars and desk objects of other students. The desk objects of other students may be desk objects synthesized by inserting images of the desk tops, or desk objects without inserting images.

[0058] The teacher avatar 400 is a CG character that conducts a lesson by reflecting the movements of the teacher 40. The student terminals 30 control the teacher avatar 400 based on the motion data received from the correspondence class server 10.

[0059] 6 is not adjusted to correspond to the position and size of the desk in the real space. A student avatar (not shown) is present in front of the desk object 500. An image is fitted onto a top plate 500A of the desk object 500 and synthesized.

[0060] The blackboard object 600 is an object used by the teacher 40 as the lesson progresses. It displays information displayed on the classroom display 11 and information input from the teacher's terminal 13.

[0061] 7 is a diagram showing an example of a student's desk in real space. Desk 700 is within reach of a student watching a lesson. While the student is watching the lesson content, student terminal 30 extracts the area of ​​table top 700A of desk 700 from an image of desk 700 captured by camera 32.

[0062] Fig. 8 is a diagram showing an example of a virtual space in which an image of the desk in Fig. 7 is inserted into the virtual space in Fig. 6 and synthesized. In Fig. 8, an image within the area of ​​the tabletop 700A in Fig. 7 is inserted into the tabletop 500A of the desk object 500 in the virtual space and synthesized. In addition, the position of the tabletop 500A is slightly higher, and the size of the tabletop 500A is adjusted to match the tabletop 700A in the real space. A personal computer and a notebook placed on the desk 700 in the real space are also projected onto the desk object in the virtual space. A frame surrounding the tabletop 500A synthesized by inserting the image of the real space may be displayed.

[0063] In addition to fitting and synthesizing an image of the real world onto a desk object, for example, the screen of a computer monitor in the real world may be projected onto an object in the virtual space. Specifically, the computer keyboard and mouse are placed on the desk so that they can be operated. The computer monitor is photographed with a camera. A monitor object corresponding to the computer monitor is placed in the virtual space. The display area of ​​the monitor is extracted from the photographed image of the monitor in the real space and is fitted and synthesized into the monitor object in the virtual space. Students can check the display reflected on the monitor object in the virtual space while operating the computer in the real space. Since the students do not touch the monitor object, the position and size in the virtual space may be set arbitrarily.

[0064] In addition, a spherical camera capable of capturing 360-degree images of the student's own room can be used to capture the images, and the captured images can be embedded and composited onto the walls, floor, and ceiling of the virtual classroom.

[0065] As described above, the viewing system for lesson contents of this embodiment includes a correspondence class server 10 that distributes lessons held in a virtual space and a student terminal 30 for viewing the lessons. The correspondence class server 10 includes a distribution unit 105 that distributes virtual space information for displaying lessons held in a virtual space. The student terminal 30 includes a VR function unit 301 that displays the virtual space based on the virtual space information, and an input unit 302 that inputs an image of a real space within reach of a student viewing the lesson. The VR function unit 301 extracts an area in which the top plate 700A of the desk 700 is shown from the image, and combines the image within the area with the top plate 500A of the desk object 500. This allows the student to see the desk used by the student without removing the HMD 31, and further allows the student to use textbooks, notebooks, computers, etc. on the desk. [Explanation of symbols]

[0066] 10…Server for correspondence classes 101…Input / output section 102...Control unit 103...Audio processing unit 104...Storage section 105…Distribution Department 11…Classroom display 12...Broadcast camera 13…Teacher terminal 14...Microphone 15…Speaker 20…Communication lines 30…Student terminals 301…VR function section 302... Input section 303…Measurement part 304...Adjustment section 305...Storage section 31...HMD 32…Camera

Claims

1. A viewing terminal for viewing content in a virtual space, a head mounted display including a camera and displaying the virtual space based on virtual space information; A measurement unit that measures three-dimensional space information of a real space; an input unit for inputting an image of the real space captured by the camera, extracting a target area from the image based on the three-dimensional space information, fitting and synthesizing the image within the target area into an object displayed in the virtual space, placing an avatar of a viewer in the virtual space, and making the avatar's hand or arm invisible or transparent if the hand or arm of a user is reflected in the image within the target area; The positional relationship between the viewpoint and the object in the virtual space is determined based on the positional relationship between the head mounted display and the target area in the real space. A viewing terminal characterized by:

2. 2. The viewing terminal according to claim 1, A planar area corresponding to the target area is extracted from the image of the real space, and the image of the planar area is fitted into the object in the virtual space to synthesize the object. A viewing terminal characterized by:

3. A viewing terminal for viewing content in a virtual space, a head mounted display that displays the virtual space based on virtual space information; A camera that captures real space, an input unit for inputting an image of the real space, The image of the object in the target area in the real space is fitted to the object displayed in the virtual space to synthesize the object. a positional relationship between a viewpoint and the object in the virtual space is determined based on a positional relationship between the head mounted display and the target area in a real space; A camera capturing the real space and an object having the target area corresponding to the object are fixed. A viewing terminal characterized by:

4. A viewing method for viewing content in a virtual space using a viewing terminal, comprising: displaying the virtual space based on virtual space information; Measuring three-dimensional space information of a real space; and inputting an image of the real space captured by a camera of the head mounted display, In the displaying step, a target area is extracted from the image based on the three-dimensional space information, the image in the target area is fitted and synthesized with an object displayed in the virtual space, an avatar of a viewer is placed in the virtual space, and if a hand or arm of a user is reflected in the image in the target area, the hand or arm of the avatar is made invisible or transparent; The positional relationship between the viewpoint and the object in the virtual space is determined based on the positional relationship between the head mounted display and the target area in the real space. A viewing method comprising:

5. A program for operating a computer as a viewing terminal for viewing content in a virtual space, A process of displaying the virtual space based on virtual space information; A process of measuring three-dimensional spatial information of a real space; inputting an image of the real space captured by a camera of a head-mounted display; In the display process, a target area is extracted from the image based on the three-dimensional space information, the image in the target area is fitted and synthesized with an object displayed in the virtual space, an avatar of a viewer is placed in the virtual space, and when a user's hand or arm is reflected in the image in the target area, the computer is caused to execute a process of making the hand or arm of the avatar invisible or transparent; A program in which the positional relationship between the viewpoint and the object in the virtual space is determined based on the positional relationship between the head-mounted display and the target area in real space.

Images