Virtual space control system, virtual space control method, and program

Abstract

To enable the use of real-world tangible objects while wearing an HMD.SOLUTION: A student terminal 30 for viewing classes conducted in an immersive virtual space comprises: a VR function unit 301 that displays a virtual space based on virtual space information; and an input unit 302 that receives images capturing the real space within the reach of a student viewing the class. The VR function unit 301 extracts, from the images, a region where the top surface of a desk corresponding to a desk object in the virtual space is depicted, and embeds and composites the image within the region onto the top portion of the desk object.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technology for viewing a lesson held in a virtual space. [Background technology]

[0002] In recent years, distance learning systems using not only television and radio but also information and communication technology (ICT) have become widespread.

[0003] One possible distance learning system using ICT is to use immersive virtual reality (VR) technology, where students wear head-mounted displays (HMDs) and take classes in a virtual space. By participating in classes in an immersive virtual space, students can shut out information from the outside world and concentrate more on the class than when using television or a personal computer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-203786 [Patent Document 2] Japanese Patent Application Publication No. 2019-101945 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in an immersive virtual space where the field of view is obscured by an HMD or other device, physical objects on a desk in the real world, such as textbooks, notebooks, and computers, cannot be accessed unless the HMD is removed, making it cumbersome to put on and take off the HMD during classes held in a virtual space.

[0006] A technology is known that displays images captured by a camera in an HMD in a virtual space when it detects that the user wearing the HMD is looking downward. This technology allows users to use textbooks and notebooks in the real world without removing the HMD. However, conventional methods do not take into account objects placed in the virtual space, and instead display images of the real world below the virtual space, which can disrupt the immersive experience.

[0007] Patent Document 1 discloses a technology that combines a virtual image of a character with a real image, and makes the character move in a realistic manner that matches the figure included in the real image. The technology in Patent Document 1 combines a virtual character with real space, but does not combine an image of an actual object from real space with a virtual space.

[0008] Patent Document 2 discloses a technology for inputting handwritten characters in a virtual space. The technology in Patent Document 2 displays an object in the virtual space that corresponds to an operation panel (acrylic panel) in real space. When a user writes characters on the operation panel with their finger in real space, the written characters are recognized and displayed on an object in the virtual space. The user can input handwritten characters into the virtual space using the operation panel in real space, but the characters written on the operation panel in real space are not saved and cannot be used to create notes in real space.

[0009] The present invention has been made in view of the above, and has as its object to enable the use of actual objects in real space while wearing an HMD. [Means for solving the problem]

[0010] A virtual space control system according to one embodiment of the present invention comprises a camera, a head-mounted display, and a control unit that detects real objects from real-space images captured by the camera and displays a virtual space on the head-mounted display in which the real objects are fitted into virtual objects. [Effects of the Invention]

[0011] According to the present invention, it is possible to use actual objects in real space while wearing an HMD. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of the overall configuration of a lesson content viewing system according to this embodiment. [Figure 2] FIG. 2 is a conceptual diagram showing a classroom from which lesson content is distributed. [Figure 3] FIG. 3 is a functional block diagram showing an example of the configuration of a correspondence class server. [Figure 4] FIG. 4 is a functional block diagram showing an example of the configuration of a student terminal. [Figure 5] FIG. 5 is a flowchart showing the flow of processing for fitting an image of a desk in real space into a desk object in virtual space. [Figure 6] FIG. 6 is a diagram showing an example of a virtual space before an image is embedded and synthesized. [Figure 7] FIG. 7 is a diagram showing an example of a student's desk in the real space. [Figure 8] FIG. 8 is a diagram showing an example of a virtual space obtained by inserting the image of the desk shown in FIG. 7 into the virtual space shown in FIG. 6 and combining the images. DETAILED DESCRIPTION OF THE INVENTION

[0013] [System Configuration] A lesson content viewing system according to one embodiment of the present invention will be described below with reference to the drawings.

[0014] The lesson content viewing system of this embodiment is a system that distributes lesson content using virtual reality (VR). The lesson content viewing system shown in Fig. 1 includes a correspondence class server 10 and a student terminal 30. The correspondence class server 10 and the student terminal 30 are connected via a communication line 20. The communication line 20 is any communication line such as the Internet.

[0015] Although only one student terminal 30 is shown in FIG. 1, the number of student terminals 30 can be any number. One student terminal 30 is provided for each student. When lesson content is distributed by live broadcast, many student terminals 30 are simultaneously connected to the distance learning server 10. When lesson content is distributed on demand, student terminals 30 and distance learning server 10 may be connected one-to-one.

[0016] Any computer device, such as a smartphone or personal computer, can be used as the student terminal 30, as long as it has a VR function. An HMD 31 and a camera 32 are connected to the student terminal 30. The camera 32 may be provided in the HMD 31 or may be fixed in the room where the student is present. A device in which the student terminal 30, HMD 31, and camera 32 are integrated may also be used.

[0017] When viewing the lesson content, students wear HMD 31. Student terminal 30 receives information for constructing a virtual space from distance learning server 10, constructs the virtual space based on the received information, and displays a video of the virtual space on HMD 31. When rendering the virtual space, student terminal 30 embeds an image of the student's desk, captured by camera 32, into a desk object in the virtual space and composites it.

[0018] Connected to the correspondence class server 10 are a classroom display 11, a relay camera 12, a teacher's terminal 13, a microphone 14, and a speaker 15. The speaker 15 includes headphones (earphones). The correspondence class server 10 creates lesson content in a virtual space based on information received from the connected devices.

[0019] As shown in FIG. 2, teacher 40 stands in front of classroom display 11 and uses classroom display 11 and teacher terminal 13 to teach a class. Classroom display 11 is a large-screen display known as an electronic whiteboard. Teacher 40 teaches by displaying lesson data stored in correspondence class server 10 on classroom display 11. The lesson data may be displayed on teacher terminal 13. The lesson progresses as the lesson data changes and is displayed in accordance with teacher 40's utterances. Teacher 40 may operate teacher terminal 13 to select the lesson data to be displayed on classroom display 11.

[0020] The lesson data displayed on the classroom display 11 is also displayed on a blackboard object located in the virtual space. A blackboard or whiteboard may be used instead of the classroom display 11.

[0021] The relay camera 12 captures a lesson taught by the teacher 40. Video data of the teacher 40 is sent to the correspondence class server 10. The correspondence class server 10 detects the movements and posture of the teacher 40 from the video data and generates motion data to be reflected in a teacher avatar placed in a virtual space. The teacher avatar is a computer graphics (CG) character that reflects the movements of the teacher 40. For example, the correspondence class server 10 detects the bones (skeleton) of the teacher 40 from the video data and moves the teacher avatar based on the detected bones. The correspondence class server 10 distributes the motion data of the teacher avatar generated based on the bones to the student terminals 30. Instead of creating motion data of the teacher avatar from video data of the teacher 40, the teacher may wear an HMD and operate a controller to control the teacher avatar.

[0022] The microphone 14 picks up the voice uttered by the teacher 40. The picked-up voice data is sent to the correspondence class server 10. The correspondence class server 10 distributes the voice data to the student terminals 30.

[0023] The speaker 15 outputs, for example, instructions from the filming staff regarding the progress of the lesson to the teacher 40. If the lesson is two-way, the speaker 15 may output audio data transmitted from the student terminals 30.

[0024] It should be noted that any video footage of a lesson can be used to create lesson content. For example, the bones of characters in a previously recorded educational program can be detected and avatars in a virtual space can be moved based on the detected bones.

[0025] [Server Configuration] The configuration of the correspondence class server 10 will be described with reference to Fig. 3. The correspondence class server 10 shown in Fig. 3 comprises an input / output unit 101, a control unit 102, an audio processing unit 103, a storage unit 104, and a distribution unit 105.

[0026] The input / output unit 101 is connected to the classroom display 11, relay camera 12, teacher terminal 13, microphone 14, and speaker 15, and outputs lesson data and inputs video data, audio data, and operations.

[0027] The control unit 102 detects the movements and postures of the teacher 40 from the video data and generates motion data for the teacher avatar. The motion data is stored in the memory unit 104. The control unit 102 stores in the memory unit 104 the information displayed on the classroom display 11 and the information input from the teacher terminal 13.

[0028] The audio processing unit 103 stores audio and the like received by the microphone 14 in the storage unit 104 .

[0029] The memory unit 104 stores information for constructing a virtual space in which a lesson is held. Examples of information for constructing a virtual space include model data of objects to be placed in the virtual space, object placement information, and avatar model data and motion data. Examples of objects to be placed in the virtual space include a blackboard object, classroom wall, floor, and ceiling objects, and student desk objects. The memory unit 104 stores lesson data, including audio data recorded as the lesson progresses, information displayed on the classroom display 11, and information input from the teacher's terminal 13.

[0030] The distribution unit 105 distributes virtual space information for constructing a virtual space in which a lesson will be held. For example, before the start of a lesson, the distribution unit 105 distributes model data of objects and avatars placed in a classroom in the virtual space to the student terminals 30. While the lesson is being viewed, the distribution unit 105 distributes data that changes according to the progress of the lesson, such as motion data of the teacher avatar, audio data, and information drawn on a blackboard object, to the student terminals 30 as needed.

[0031] [Device configuration] The configuration of the student terminal 30 will be described with reference to Fig. 4. The student terminal 30 shown in Fig. 4 includes a VR function unit 301, an input unit 302, a measurement unit 303, an adjustment unit 304, and a storage unit 305.

[0032] The VR function unit 301 receives virtual space information for constructing a virtual space from the correspondence class server 10 and constructs the virtual space. The VR function unit 301 controls the virtual camera based on the student's head movement detected by the HMD 31 and renders the virtual space. The image seen by the student is the virtual space seen from the viewpoint of the student avatar in the virtual space. The student avatar is a CG character in the virtual space of the student watching the class.

[0033] When rendering the virtual space, the VR function unit 301 analyzes the image of the real space captured by the camera 32, extracts the area of ​​the top of the desk from the image, and composites the image of the extracted area onto the top of the desk object in the virtual space. As a result, the image of the desk in the real space is blended with the desk object in the virtual space and displayed. Because everything except the top of the desk object into which the image is composited is an image rendered from the virtual space, the sense of immersion is not lost.

[0034] By looking at the desk object as a student avatar, that is, the student, while wearing the HMD 31, can see the desk in real space projected onto the top of the desk object. In the virtual space, the student can see the textbooks, notebooks, computers, etc. on the desk in real space. Since the characters written by the student in the notebook in real space are projected onto the desk object, the student can create notes in real space while wearing the HMD 31. The image of real space captured by the camera 32 is an image of the real space that includes at least the area within the reach of the student. The student can use real-space objects while viewing the image of the real space merged with the virtual space. Note that the area within the student's reach extracted from the image may be inserted and synthesized into the virtual space other than the desk object.

[0035] When a student opens a desk drawer, the VR function unit 301 may extract the area of ​​the drawer from within the video, set the desk object to the open drawer state, and then fit the area of ​​the drawer into the desk object to synthesize the image. The desk into which the image is fitted and synthesized is not limited to a desk, but may also be a piece of furniture such as a side table, a side wagon, or a side chest. The desk object also includes objects corresponding to these pieces of furniture.

[0036] The VR function unit 301 may generate motion data of the student avatar in the virtual space and transmit it to the correspondence class server 10. The student may control the motion of the student avatar by operating a controller.

[0037] The input unit 302 inputs images captured by the camera 32. The input unit 302 may also input the student's head movements and controller operations detected by the HMD 31. A microphone may be connected to the input unit 302 to input the student's voice, and the input voice data may be transmitted to the correspondence class server 10.

[0038] Before the student watches the lesson content, the measurement unit 303 measures three-dimensional spatial information of the real space where the student's desk is located, and detects the desk top in the real space based on the obtained three-dimensional spatial information. For example, the measurement unit 303 can measure the three-dimensional spatial information of the shooting location by moving the camera 32 and using markerless AR technology with a monocular camera. For example, based on the obtained three-dimensional spatial information, the measurement unit 303 detects a flat surface with a predetermined height and a predetermined area as the desk top.

[0039] The measurement unit 303 may present several planes that are candidates for the tabletop to the student, and the student may select the plane that indicates the tabletop. Specifically, for example, the student wears the HMD 31 equipped with the camera 32, and an image captured by the camera 32 is displayed on the HMD 31. The student moves their head (camera 32) so that the desk top fits within the frame of the camera 32. The measurement unit 303 measures three-dimensional spatial information from the image to detect the plane. When the measurement unit 303 detects the plane, it displays a frame indicating the detected plane superimposed on the image captured by the camera 32. When the frame indicating the plane matches the desk top, the student performs a decision operation, such as operating a controller.

[0040] The adjustment unit 304 adjusts the position of the desk top of the desk object in the virtual space based on the positional relationship between the HMD 31 and the desk top in the real space. The position of the HMD 31 corresponds to the head of the student avatar. The adjustment unit 304 may adjust the size of the desk top of the desk object to the size of the desk top in the real space. By adjusting the position and size of the desk object by the adjustment unit 304 to match the desk in the real space, when a student touches the desk top of the desk object with the student avatar, the student can also touch the desk top in the real space. The size of the student avatar is adjusted to suit the student. Objects that can be touched in the real space are displayed in the virtual space.

[0041] The storage unit 305 holds information necessary for the VR function unit 301 to render the virtual space, three-dimensional space information measured by the measurement unit 303, and information on the desk object after adjustment by the adjustment unit 304. The storage unit 305 may also store information input by the input unit 302, such as video captured by the camera 32.

[0042] When preparing a dedicated booth for students to view lesson content, the camera 32 may be fixedly positioned so as to capture the desks used by the students. Desks of a predetermined size are used and placed in predetermined positions. As a result, the positions of the camera 32 and the desks are fixed, so there is no need for the measurement unit 303 to detect the table top. As the size of the desk is also predetermined, there is no need for the adjustment unit 304 to adjust the desk object. The camera 32 may also be fixedly positioned in the student's own room. Once the desk object has been adjusted to match the desk in the student's room, no adjustment is required from the second time onwards.

[0043] For example, a general-purpose computer system equipped with a central processing unit (CPU), memory, storage, communication devices, and input / output devices can be used for the distance learning server 10 and the student terminals 30. In this computer system, the CPU executes a predetermined program loaded into the memory, thereby realizing the distance learning server 10 and the student terminals 30. This program can be recorded on a computer-readable recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

[0044] The distance learning server 10 may render the virtual space. For example, the student terminal 30 transmits the student's head movement detected by the HMD 31 to the distance learning server 10. The distance learning server 10 controls the virtual camera based on the student's head movement and renders the virtual space. The student terminal 30 receives and displays the rendered image.

[0045] [Inlay synthesis processing] The flow of the process of performing the embedding synthesis will be described with reference to FIG.

[0046] In step S11, the measurement unit 303 detects the position and size of the desk in the real space.

[0047] In step S12, the adjustment unit 304 adjusts the position and size of the desk object in the virtual space based on the position and size of the desk in the real space. For example, the adjustment unit 304 adjusts the distance from the viewpoint of the student avatar to the desk top based on the distance from the HMD 31 to the desk top, and further adjusts the size of the desk top.

[0048] After the desk object is adjusted by the adjustment unit 304, the student may wear the HMD 31 to display the desk object in the virtual space, and move his / her hands to check whether the position and size of the desk object in the virtual space correspond to the position and size of the desk in the real space, and fine-tune the position and size of the desk object.

[0049] By the above processing of steps S11 and S12, the desk object in the virtual space can be made to correspond to the desk in the real space.

[0050] In step S13, the area on the desk is extracted from the image captured by camera 32, and the image of the extracted area is fitted onto the top of the desk object in the virtual space and composited. This allows the student to view the desk in real space within the virtual space while wearing HMD 31.

[0051] When a student writes in a notebook on a desk in real space, the student's hands and arms appear in the image of the desk. In this case, the student's hands and arms may be hidden or made transparent.

[0052] When rendering an object, the VR function unit 301 may highlight a portion that corresponds to an actual object in real space. For example, for a desk top that corresponds to an actual object in real space, the VR function unit 301 highlights the edge of the desk top by surrounding it with a frame. This allows students to recognize that the portion surrounded by the frame in the virtual space is an actual object in real space.

[0053] To synthesize an image of real space without shaking relative to an object even when a student's head shakes.

[0054] The function for embedding and compositing real-world images may be turned on or off, or the transparency of the real-world images may be set and embedded. Even if the real-world images are not embedded and composited, the position and size of the desk object in the virtual space should correspond to the desk in the real world.

[0055] [Example of inlay synthesis] An example of compositing an image into a desk object will be described with reference to FIGS.

[0056] 6 is a diagram showing an example of a virtual space before video is embedded and synthesized. The diagram shows a teacher avatar 400, a desk object 500, and a blackboard object 600. Objects other than those shown, such as floors, ceilings, and walls, may also be placed in the virtual space.

[0057] When multiple students are taking a class together or when creating the illusion of multiple students taking a class together, avatars and desk objects of other students may be placed. The desk objects of other students may be desk objects synthesized by inserting images of the desk tops, or they may be desk objects without inserting images.

[0058] The teacher avatar 400 is a CG character that conducts the lesson by reflecting the movements of the teacher 40. The student terminals 30 control the teacher avatar 400 based on the motion data received from the correspondence class server 10.

[0059] 6 has not been adjusted to correspond to the position and size of the desk in real space. A student avatar (not shown) exists in front of the desk object 500. An image is embedded in and composited onto the top plate 500A of the desk object 500.

[0060] The blackboard object 600 is an object used by the teacher 40 as the lesson progresses. It displays information displayed on the classroom display 11 and information input from the teacher's terminal 13.

[0061] 7 is a diagram showing an example of a student's desk in real space. Desk 700 is a desk within reach of a student watching a lesson. While the student is watching the lesson content, student terminal 30 extracts the area of ​​desk top 700A of desk 700 from the video of desk 700 captured by camera 32.

[0062] Fig. 8 is a diagram showing an example of a virtual space obtained by embedding a captured image of the desk of Fig. 7 into the virtual space of Fig. 6. In Fig. 8, an image within the area of ​​tabletop 700A of Fig. 7 is embedded into tabletop 500A of desk object 500 in the virtual space and composited. In addition, the position of tabletop 500A is slightly higher, and the size of tabletop 500A is adjusted to match tabletop 700A in the real space. A personal computer and notebook placed on desk 700 in the real space are also projected onto the desk object in the virtual space. A frame may be displayed surrounding tabletop 500A, which has been composited by embedding the image of the real space.

[0063] In addition to fitting and synthesizing a real-world image onto a desk object, it is also possible to project, for example, the screen of a computer monitor in real space onto an object in virtual space. Specifically, the computer keyboard and mouse are placed on the desk so that they can be operated. A photograph of the computer monitor is taken with a camera. A monitor object corresponding to the computer monitor is placed in virtual space. The monitor display area is extracted from the photographed image of the real-world monitor and fitted into the monitor object in virtual space for synthesis. Students can check the display reflected on the monitor object in virtual space while operating the computer in real space. Because the students do not touch the monitor object, its position and size in virtual space can be set as desired.

[0064] It is also possible to take a picture of the student's room with a spherical camera capable of taking 360-degree images, and then embed the captured image onto the walls, floor, and ceiling of the virtual classroom.

[0065] As described above, the lesson content viewing system of this embodiment includes a correspondence class server 10 that distributes lessons held in a virtual space and a student terminal 30 for viewing the lessons. The correspondence class server 10 includes a distribution unit 105 that distributes virtual space information for displaying lessons held in the virtual space. The student terminal 30 includes a VR function unit 301 that displays the virtual space based on the virtual space information and an input unit 302 that inputs video captured of the real space within reach of the student viewing the lesson. The VR function unit 301 extracts an area showing the tabletop 700A of the desk 700 from the video and then embeds the video within the area onto the tabletop 500A of the desk object 500 to create a composite image. This allows the student to view the desk they are using without removing the HMD 31 and also use textbooks, notebooks, computers, and other items on the desk. [Explanation of symbols]

[0066] 10. Correspondence class server 101…Input / output section 102...Control unit 103...Audio processing unit 104...Storage section 105…Distribution Department 11...Classroom display 12...Broadcast camera 13...Teacher's terminal 14...Microphone 15...Speaker 20...Communication lines 30...Student devices 301...VR function section 302...input section 303…Measurement part 304...Adjustment section 305...Storage section 31...HMD 32...Camera

Claims

1. A camera and A head-mounted display and a control unit that detects a real object from the real space image captured by the camera, and displays a virtual space in which the real object is fitted into a virtual object on the head-mounted display. Virtual space control system.

2. 2. The virtual space control system according to claim 1, the control unit measures three-dimensional space information of a real space based on a markerless AR technology, and detects the real object based on the three-dimensional space information. Virtual space control system.

3. 3. The virtual space control system according to claim 1, the real object is an object having a plane; Virtual space control system.

4. 4. The virtual space control system according to claim 1, the control unit displays a virtual space that does not include a real space image other than the virtual object into which the real object is embedded and synthesized. Virtual space control system.

5. 5. A virtual space control system according to claim 1, the control unit detects a position of the real object and adjusts a position of the virtual object based on the position of the real object. Virtual space control system.

6. 6. A virtual space control system according to claim 1, the control unit detects a size of the real object and adjusts a size of the virtual object based on the size of the real object. Virtual space control system.

7. 7. A virtual space control system according to claim 1, the control unit places an avatar of the user wearing the head-mounted display in the virtual space, controls a virtual camera based on the movement of the user's head, and displays the virtual space as seen from the viewpoint of the avatar. Virtual space control system.

8. Detect real objects from real-world images captured by a camera, a virtual space in which the real object is fitted into the virtual object and synthesized is displayed on a head-mounted display; Virtual space control method.

9. On the computer, A process for detecting real-world objects from real-world images captured by a camera; a process of displaying, on a head-mounted display, a virtual space in which the real object is fitted into the virtual object and synthesized; A program that executes the following.

Images