Real-space projection method, information processing device, information processing system, and program

Abstract

To provide a real-space projection method, information processing device, information processing system, and program that can project one's own or another's actions into real space. [Solution] A real-space projection method for projecting an object executed by a computer into real space, comprising the steps of: photographing a person including oneself; extracting motion data from the photographed image of the person; and projecting either the motion of the person or the motion of another person, or both, into real space based on the extracted motion data, wherein the projected motion of the person is objectively observable from the perspective of the person.

Description

Technical Field

[0001] Embodiments of the present invention relate to a real-space projection method, an information processing apparatus, an information processing system, and a program.

Background Art

[0002] Techniques for estimating the human skeleton from motion capture and displaying an avatar, operating an effect, and making them observable based on the data are known (see, for example, Patent Document 1). In addition, a system for assisting in improving operations by comparing videos of an instructor and a learner is known (see, for example, Patent Document 2). Further, a method for a user to switch viewpoints within a 360-degree video is known (see, for example, Patent Document 3). Also, a method for displaying an effect or the like for appropriately visually recognizing information that is likely to be overlooked by the user through gaze guidance is known (see, for example, Patent Document 4).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technology described in Patent Document 1 is primarily designed to improve theatrical and gaming experiences, and its application in the entertainment field is emphasized. Therefore, it cannot promote learning effects by observing the actions of people, including oneself, as the object. The technology described in Patent Document 2 allows observation only from a pre-recorded viewpoint and cannot be observed by projecting it into real space using augmented reality or mixed reality technologies. The technology described in Patent Document 3 cannot switch viewpoints using virtual reality or mixed reality. The technology described in Patent Document 4 cannot highlight parts of a person's actions, including one's own, that should be focused on for learning purposes.

[0005] The object of the present invention is to provide a real-space projection method, information processing device, information processing system, and program that can project either or both of the actions of a person including oneself and the actions of other people into real space. [Means for solving the problem]

[0006] (1) One embodiment of the present invention is a real-space projection method for projecting an object executed by a computer into real space, comprising the steps of: photographing a person including oneself; extracting motion data from the image of the person photographed; and projecting either the motion of the person or the motion of another person, or both, into real space based on the extracted motion data, wherein the projected motion of the person is objectively observable from the self.

[0007] (2) One embodiment of the present invention is the real space projection method described above, wherein in the projection step, the projection is performed in real space at any timing.

[0008] (3) One embodiment of the present invention is the real space projection method described above, in which, in the projection step, if the movement of the person is projected, the projected movement of the person is projected into the real space together with a predetermined desired movement of the person.

[0009] (4) One embodiment of the present invention provides a mixed reality or augmented reality in the above-described real space projection method, wherein the projection step provides a mixed reality or augmented reality.

[0010] (5) One embodiment of the present invention further comprises the above-described real-space projection method, which, instead of projecting either the actions of the person and the actions of the other person or both into real space, projects the actions of the other person as seen from the viewpoint of the person being projected into real space into real space.

[0011] (6) One embodiment of the present invention is a real space projection method described above, further comprising the steps of detecting a gesture input of the person, and, based on the gesture input of the person detected in the detection step, projecting into real space the actions of the other person as seen from the viewpoint of the person being projected into real space, instead of projecting into real space either the actions of the person or the actions of the other person or both of them.

[0012] (7) One embodiment of the present invention further comprises the above-described real-space projection method, which, instead of projecting the actions of other persons as seen from the viewpoint of the person projected in real space, projects either or both of the actions of the person and the actions of other persons into real space based on the action data extracted from the captured image of the person.

[0013] (8) One embodiment of the present invention is the real space projection method described above, wherein in the projection step, the actions of the person and the actions of another person, or both, as seen from an arbitrary viewpoint, are projected into real space.

[0014] (9) One embodiment of the present invention further comprises the step of projecting the movements of the person into real space based on the movement data extracted from a photographed image of the person, instead of projecting either or both of the movements of the person and the movements of other people into real space as seen from an arbitrary viewpoint.

[0015] (10) One embodiment of the present invention is a real-space projection method described above, further comprising the steps of: recognizing the voice of the person; and, instead of projecting the actions of other people as seen from the viewpoint of the person projected in real space based on the voice of the person recognized in the recognition step, projecting either the actions of the person or the actions of other people, or both, into real space based on the action data extracted from a captured image of the person.

[0016] (11) One embodiment of the present invention further comprises the above-described real-space projection method, which includes the step of comparing the projected movement of the person with a predetermined desired movement of the person.

[0017] (12) One embodiment of the present invention is the real space projection method described above, wherein in the shooting step, shooting is performed using one of the motion capture camera, RGB camera, RGBD camera, spatial video camera, or a combination thereof.

[0018] (13) One embodiment of the present invention is the real space projection method described above, in which the motion data is converted into a skeletal motion, a mesh, or a 3D model in the extraction step.

[0019] (14) One embodiment of the present invention is an information processing device comprising: an extraction unit that extracts motion data from an image of a person that has been photographed; and a projection unit that projects either or both of the motion of the person and the motion of another person into real space based on the motion data extracted by the extraction unit, wherein the projected motion of the person is objectively observable from the person.

[0020] (15) One embodiment of the present invention is an information processing system comprising: an imaging unit that photographs a person including itself; an extraction unit that extracts motion data from an image of the person taken by the imaging unit; and a projection unit that projects either or both of the person's motion and / or the motion of another person into real space based on the motion data extracted by the extraction unit, wherein the projected motion of the person is objectively observable from the self.

[0021] (16) One embodiment of the present invention is a program that causes a computer to execute steps of extracting motion data from an image of a person including the captured self, and projecting either or both of the motion of the person and the motion of another person onto the real space based on the motion data extracted in the extracting step, wherein the projected motion of the person is objectively observable from the self.

Advantages of the Invention

[0022] According to an embodiment of the present invention, it is possible to provide a real space projection method, an information processing apparatus, an information processing system, and a program that can project either or both of the motion of a person including the self and the motion of another person onto the real space.

Brief Description of the Drawings

[0023] [Figure 1] It is a diagram showing a configuration example of an information processing system 1 according to an embodiment of the present invention. [[ID=第十九]] [[ID=第二十]] [Figure 2] [[ID=第二十一]]It is a diagram showing an example of an image of a person captured. [[ID=第二十二]] [[ID=第二十三]] [Figure 3A] [[ID=第二十四]]It is a diagram for explaining an example of the operation of an information processing apparatus 100 according to the present embodiment. [[ID=第二十五]] [[ID=第二十六]] [Figure 3B] [[ID=第二十七]]It is a diagram for explaining an example of the operation of an information processing apparatus 100 according to the present embodiment. [[ID=第二十八]] [[ID=第二十九]] [Figure 4] [[ID=第三十]]It is a diagram for explaining an example of the operation of an information processing apparatus 100 according to the present embodiment. [[ID=第三十一]] [[ID=第三十二]] [Figure 5] [[ID=第三十三]]It is a configuration diagram showing an example of a communication apparatus 300 according to the present embodiment. [[ID=第三十四]] [[ID=第三十五]] [Figure 6] [[ID=第三十六]]It is a diagram for explaining an image corresponding to the pointing direction. [[ID=第三十七]] [[ID=第三十八]] [Figure 7] [[ID=第三十九]]It is a flowchart showing an example of the operation of an information processing system 1 according to the present embodiment. [[ID=第四十]] [[ID=第四十一]] [Figure 8] [[ID=第四十二]]It is a diagram showing a configuration example of an information processing system 1 of Modification Example 1 of the embodiment. [[ID=第四十三]] s>[[ID=第四十四]] [Figure 9]This is a configuration diagram showing an example of a communication device 500 according to this embodiment. [Figure 10] This flowchart shows an example of the operation of the information processing system 1 according to a modified embodiment. [Figure 11] This figure illustrates an example of the operation of an information processing system 1 according to a modified embodiment. [Figure 12] This figure shows an example of the operation of the information processing system 1 according to a modified embodiment. [Figure 13] This figure shows an example configuration of the information processing system 1 in a modified example of the embodiment 2. [Figure 14] This flowchart shows an example of the operation of the information processing system 1 according to a modified example of the embodiment 2. [Figure 15] This figure shows an example of the operation of the information processing system 1 according to a modified example 2 of the embodiment. [Modes for carrying out the invention]

[0024] Next, the real-space projection method, information processing device, information processing system, and program of this embodiment will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below. In all the figures used to illustrate the embodiments, components with the same function are given the same reference numerals, and repeated explanations are omitted. Furthermore, in this application, "based on XX" means "based on at least XX," and includes cases where it is based on another element in addition to XX. Also, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on something that has been calculated or processed from XX. "XX" is any element (for example, any information).

[0025] (Embodiment) (Information Processing System) Figure 1 shows an example configuration of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes an information processing device 100, an imaging unit 200, and a communication device 300. The information processing device 100, the imaging unit 200, and the communication device 300 each communicate with each other via a network NW. The network NW includes at least one of the following: the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), a mobile communication network, a cellular network, etc.

[0026] Between the information processing device 100, the imaging unit 200, and the communication device 300, the movements of a person created by the information processing device 100 based on at least a portion of the image of the person captured by the imaging unit 200 are projected (displayed) by the communication device 300, thereby providing the communication device 300 with mixed reality (MR) or augmented reality (AR).

[0027] The information processing device 100 receives an image of a person captured by the imaging unit 200. Here, the person includes the user. The information processing device 100 extracts motion data representing the person's movements from the image of the person. Based on the extracted motion data of the person, the information processing device 100 creates projection information to project the person's movements into real space. The information processing device 100 transmits the created projection information to the communication device 300.

[0028] The communication device 300 receives projection information from the information processing device 100 and projects the movements of a person into real space based on the received projection information. The communication device 300 projects the movements of the person so that they can be objectively observed from the user's own perspective. Here, "the user" refers to a person who is provided with mixed reality or augmented reality from the communication device 300 and who can observe the movements of the projected person. The other person (other) may be one person or multiple people. Hereinafter, the user may be referred to as the first user U1, and the other person (other) may be referred to as the second user U2.

[0029] (Information processing device 100) The information processing device 100 is implemented by a device such as a personal computer, server, smartphone, tablet computer, or industrial computer. The information processing device 100 includes, for example, a communication unit 101, an extraction unit 102, a projection unit 103, and a storage unit 110.

[0030] The communication unit 101 is implemented by a communication module. The communication unit 101 communicates with communication devices such as the imaging unit 200 and the communication device 300 via a network NW. The imaging unit 200 is assumed to have a communication function. The communication unit 101 may communicate using a communication method such as wired LAN. Alternatively, the communication unit 101 may communicate using a wireless communication method such as wireless LAN, Bluetooth®, or LTE®. The communication unit 101 receives an image of a person transmitted by the imaging unit 200. Figure 2 shows an example of an image of a person. Figure 2 shows an image of a person practicing basketball.

[0031] The extraction unit 102 acquires an image of a person from the communication unit 101. The extraction unit 102 extracts motion data representing the person's movements from the acquired image of the person. Specifically, the extraction unit 102 extracts motion data by converting the person's body from the image of the person into a skeletal movement, mesh, or 3D model.

[0032] Figures 3A and 3B are diagrams illustrating an example of the operation of the information processing device 100 according to this embodiment. Referring to Figures 3A and 3B, the process of extracting motion data of a person's body from an image of a person will be described. As shown in Figure 3A, the extraction unit 102 detects the part of the person from the image of the person captured and sets the region containing the detected part of the person (hereinafter referred to as the "person detection region PDA"). From the set person detection region PDA, the extraction unit 102 extracts detailed motion data such as the position and angle of the person's joints, the speed of movement, and acceleration, and estimates the person's posture.

[0033] The extraction unit 102 may also perform three-dimensional coordinate analysis. By performing three-dimensional coordinate analysis, the motion data includes the position and angle of the joints of the person included in the person's image, the velocity of movement, acceleration, etc., derived based on three-dimensional coordinates. Below, we will continue the explanation as an example of the case where the extraction unit 102 performs three-dimensional coordinate analysis. The extraction unit 102 extracts an animation representing the person based on the extracted motion data and the estimated posture of the person. Figure 3B is a diagram showing an example of an animation representing the person extracted from an image of the person shown in Figure 3A. Returning to Figure 1, we will continue the explanation.

[0034] The projection unit 103 acquires motion data from the extraction unit 102. Based on the acquired motion data, the projection unit 103 creates projection information for projecting the person's movements into real space. An example of a person's movement may be the same as the person's movements. The projection unit 103 transmits the created projection information from the communication unit 101 to the communication device 300. Figure 4 is a diagram illustrating an example of the operation of the information processing device 100 according to this embodiment. Figure 4 shows an example of a 3D model of a person projected by the communication device 300 based on projection information. The 3D model may be an avatar. For example, the communication device 300 uses mixed reality / augmented reality technology to make the 3D model move according to the coordinates in which the person actually moved. As a result, the information processing device 100 can cause the communication device 300 to reproduce the person's movements, and the first user U1 can objectively observe the person's movements by referring to the movements of the projected 3D model. Furthermore, the projection unit 103 may create projection information for projecting the actions of other people into real space based on the acquired motion data. An example of another person's actions may be the same as the person's actions, or it may be an action that has been pre-set as the actions of another person. The projection unit 103 may be configured to create projection information for projecting either one or both of the actions of a person and / or the actions of another person into real space.

[0035] (Imaging unit 200) The imaging unit 200 is a camera that captures real space and generates an image using various components such as an image sensor, such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), and a lens for controlling the imaging of a subject onto the image sensor. The imaging unit 200 may capture still images or moving images. The imaging unit 200 may capture three-dimensional images.

[0036] The imaging unit 200 records a person's movements with high precision by imaging the person. For example, the person may be performing physical actions such as exercise. The imaging unit 200 may also be a camera system. The camera system includes motion capture cameras, RGB cameras, RGBD cameras, spatial video cameras, etc. An RGB camera captures light of red, green, and blue wavelengths (RGB) to display vivid color images of people and objects. An RGB camera uses visible light with wavelengths of 400nm to 700nm.

[0037] An RGBD camera is a type of depth camera that outputs both depth (D) and color (RGB) data in real time. Depth information is acquired through depth maps / images created by 3D depth sensors such as stereo sensors and time-of-flight sensors. The RGBD camera combines the RGB data and depth information pixel by pixel and delivers both in a single frame. The imaging unit 200 may be equipped with more than one camera. By equipping it with multiple cameras, it is possible to simultaneously capture and record images of a person from multiple viewpoints.

[0038] (Communication device 300) Figure 5 is a configuration diagram showing an example of a communication device 300 according to this embodiment. The communication device 300 includes, for example, a communication unit 310, a detection device 330, an image display device 350, an HMI (Human-machine Interface) 360, and a control device 370. The detection device 330 includes, for example, a direction detection device 332 and a head position detection device 334.

[0039] The communication unit 310 is a communication interface for communicating with the information processing device 100 via a network NW. The communication unit 310 is implemented by a communication module. The communication unit 310 communicates with other communication devices, such as the information processing device 100, via the network NW. The communication unit 310 may communicate using a communication method such as a wired LAN. Alternatively, the communication unit 310 may communicate using a wireless communication method such as wireless LAN, Bluetooth®, or LTE®.

[0040] The direction detection device 332 is a device for detecting the direction of direction. The direction of direction is the orientation of the first user U1's face or gaze, or both. For example, the direction of direction may be an angle that includes both an angle in the horizontal plane and a vertical component, or an angle that includes either an angle in the horizontal plane or a vertical component. The direction detection device 332 may include physical sensors such as an acceleration sensor and a gyroscope attached to the VR goggles 355 described later, or it may be an infrared sensor that detects multiple positions of the first user U1's head, or a camera that images the first user U1's head. In any case, the control device 370 calculates the direction of direction based on the information input from the direction detection device 332.

[0041] The head position detection device 334 is a device for detecting the position of the first user U1's head, such as its height. For example, the head position detection device 334 may be an accelerometer attached to the VR goggles 355. In this case, the control device 370 detects the position of the first user U1's head by integrating the output of the accelerometer minus the gravitational acceleration. The head position information obtained in this way is provided to the control device 370 as height information.

[0042] The image display device 350 projects (displays) an image based on projection information received from the information processing device 100. Figure 6 is a diagram illustrating an image corresponding to the direction of orientation. In the example shown in Figure 6, the VR goggles 355 include a physical sensor as a direction detection device 332 and the image display device 350. The control device 370, for example, uses the center of the head of the first user U1 or the center of the VR goggles 355 as Ω, and detects the direction the VR goggles 355 are facing as the direction of orientation φ, using a pre-calibrated direction as the reference direction.

[0043] The image display device 350 projects (displays) a projection image A2, which is within an angular range of plus or minus α centered on the directional direction φ, from among projection images A1 (which have an angle of about 240 degrees in the figure, but the field of view may be expanded by the merging process) that include a 3D model created based on projection information, toward the first user U1. Return to Figure 5 and continue the explanation.

[0044] HMI360 consists of a touch panel, a voice response device (agent device), or a switch, etc. HMI360 receives various instructions from the first user U1 to the communication device 300.

[0045] The control device 370 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the communication device 300 by executing the set of instructions. Note that all the functions of the communication device 300 shown in Figure 5 may be integrated into the VR goggles 355. The direction detection unit 372 detects the direction φ based on the output of the direction detection device 332. The head position detection unit 373 detects the height of the first user U1's head based on the output of the head position detection device 334. The head position may be expressed as three-dimensional coordinates, or the head height may simply be detected as the head position.

[0046] The image editing unit 374 acquires projection information and creates a 3D model of the person based on the acquired projection information. The image editing unit 374 creates a projection image A1 that includes the created 3D model. The image editing unit 374 performs a process to extract a projection image A2 corresponding to the directional direction φ from the projection image A1 (Figure 6). In the example in Figure 6, the directional direction φ is shown for the left-right movement of the first user U1's head, but the process may also be performed to extract an image corresponding to the directional direction for the up-down movement of the head. The image display control unit 375 displays the projected image A2, which has been extracted by the image editing unit 374, on the image display device 350. At this time, the image editing unit 374 may also display on the image display device 350 an image corresponding to the directional direction φ as viewed from the height indicated by the height information of the first user U1's head. For example, the image editing unit 374 switches and displays images according to the distance from the reference position of the first user U1's head (for example, the amount of upward movement or the amount of lateral movement).

[0047] (Operation of the information processing system) Figure 7 is a flowchart showing an example of the operation of the information processing system 1 according to this embodiment. (Step S1-1) The imaging unit 200 photographs and records people. (Step S2-1) The imaging unit 200 transmits the image of the person it has captured to the information processing device 100.

[0048] (Step S3-1) In the information processing device 100, the communication unit 101 receives an image of a person captured by the imaging unit 200. The extraction unit 102 acquires the image of the person captured by the communication unit 101 and extracts motion data from the acquired image of the person. (Step S4-1) In the information processing device 100, the projection unit 103 acquires motion data from the extraction unit 102 and creates projection information for projecting a person's movements into real space based on the acquired motion data. (Step S5-1) In the information processing device 100, the communication unit 101 acquires projection information from the projection unit 103 and transmits it from the communication unit 101.

[0049] (Step S6-1) In the communication device 300, the communication unit 310 receives projection information from the information processing device 100. The image editing unit 374 acquires the projection information from the communication unit 310, and creates a 3D model of a person by image processing the acquired projection information, and creates a projection image A1 that includes the created 3D model. The image editing unit 374 then extracts a projection image A2 corresponding to the directional direction φ from the created projection image A1. The image editing unit 374 may also extract an image corresponding to the directional direction for vertical movement of the head. The image display control unit 375 causes the image display device 350 to display the projected image A2 extracted by the image editing unit 374. This provides the first user U1 with a mixed reality or augmented reality that represents the movement of a three-dimensional model of a person. At this time, the image editing unit 374 may also cause the image display device 350 to display an image corresponding to the directional direction φ as viewed from the height indicated by the height information of the first user U1's head.

[0050] In the embodiment described above, the case in which an image of a person is input to the information processing device 100 from the imaging unit 200 was explained, but the invention is not limited to this example. For example, instead of an image of a person, a video or footage of a person may be input to the information processing device 100 from the imaging unit 200. The operation after the video or footage of a person is input to the information processing device 100 from the imaging unit 200 can be applied to the embodiment described above, so a description will be omitted. In the embodiment described above, the communication device 300 may change the scale of the three-dimensional model of the person when projecting the person's movements into real space. This allows the first user U1 to observe the person's movements from any direction.

[0051] In the embodiment described above, the communication device 300 may be able to change the projection speed when projecting a person's movements into real space. This allows the part to be viewed to be projected earlier by increasing the projection speed, and the part to be viewed more slowly by decreasing the projection speed, compared to when the projection speed cannot be changed. In the embodiment described above, the communication device 300 may be able to change the projection time when projecting a person's movements into real space. This allows, for example, if you want to check the person's movements again, you to project that part by rewinding the time.

[0052] According to the information processing system 1 of this embodiment, motion data can be extracted from images of a person obtained by photography, so that the motion of a person, the motion of another person, or both can be projected into real space in real time, or projected into real space after the person has acted. For this reason, it is possible to observe and learn about a person and / or other people from any viewpoint, without being limited to observation from a specific viewpoint or depending on the distance to the subject, as is the case with conventional methods. Furthermore, by changing the scale when projecting a 3D model of a person, it becomes possible to perform overhead observations, such as viewing the entire space from above.

[0053] (Variation 1) (Information Processing System) Figure 8 shows an example configuration of the information processing system 1 of a modified embodiment 1. The information processing system 1 includes an information processing device 400, an imaging unit 200, and a communication device 500. The information processing device 400, the imaging unit 200, and the communication device 500 each communicate with each other via a network NW. The network NW includes, for example, at least one of the following: the Internet, WAN, LAN, mobile communication network, cellular network, etc.

[0054] Between the information processing device 400, the imaging unit 200, and the communication device 500, the communication device 500 is provided with augmented reality or mixed reality by displaying the actions of a person, created by the information processing device 400 based on at least a portion of the image of the person captured by the imaging unit 200, to the communication device 500.

[0055] In addition to the functions of the information processing device 100, the information processing device 400 has the function of switching perspectives between subject and object in the real world. For example, the information processing device 100 can switch at any time between the subjective perspective of the first user U1 when acting subjectively (hereinafter referred to as the "first perspective") and the perspective when the first user U1 objectively observes its own actions (hereinafter referred to as the "second perspective"). After transmitting projection information to the communication device 500, the information processing device 400 receives either a first viewpoint switching request from the communication device 500, which requests to switch the viewpoint from the second viewpoint to the first viewpoint, or a second viewpoint switching request, which requests to switch the viewpoint from the first viewpoint to the second viewpoint.

[0056] Based on the received first viewpoint switching request, the information processing device 400 creates first projection information for projecting the actions of a person as seen from the subjective first viewpoint when the first user U1 takes action, onto real space using a three-dimensional model. Based on the received second viewpoint switching request, the information processing device 400 creates second projection information for projecting the actions of the person as seen from a second viewpoint—when objectively observing the actions of the first user U1's own 3D model—into real space using the 3D model. The information processing device 400 transmits either the first projection information or the second projection information it has created to the communication device 500. The communication device 500 receives either the first projection information or the second projection information from the information processing device 400, and projects the person's movements into real space using a three-dimensional model based on whichever of the first or second projection information it receives.

[0057] After transmitting the first projection information to the communication device 500, if the information processing device 400 receives a second viewpoint switching request from the communication device 500, it creates second projection information based on the received second viewpoint switching request. This second projection information is created to project the actions of a person as seen from a second viewpoint—when the actions of the first user U1's own 3D model are objectively observed—into real space using a 3D model, instead of projecting the actions of a person as seen from the subjective first viewpoint when the first user U1 is acting proactively onto real space using a 3D model. The information processing device 400 then transmits the created second projection information to the communication device 500. The communication device 500 receives second projection information from the information processing device 400, and based on the received second projection information, instead of projecting the actions of a person as seen from a subjective first viewpoint when the first user U1 acts proactively, onto real space using a three-dimensional model, it projects the actions of a person as seen from a second viewpoint when the actions of the first user U1's own three-dimensional model are objectively observed onto real space using a three-dimensional model.

[0058] After transmitting the second projection information to the communication device 500, if the information processing device 400 receives a first viewpoint switching request from the communication device 500, it creates first projection information based on the received first viewpoint switching request. This information is used to project the actions of the person as seen from the subjective first viewpoint when the first user U1 is actively acting onto the real space using the 3D model, instead of projecting the actions of the person as seen from the second viewpoint when objectively observing the actions of the first user U1's own 3D model onto the real space using the 3D model. The information processing device 400 then transmits the created first projection information to the communication device 500. The communication device 500 receives first projection information from the information processing device 400, and based on the received first projection information, instead of projecting the actions of the person as seen from a second viewpoint when objectively observing the actions of the first user U1's own 3D model onto real space using the 3D model, it projects the actions of the person as seen from a subjective first viewpoint when the first user U1 is acting proactively onto real space using the 3D model.

[0059] (Information processing device 400) The information processing device 400 is implemented by a device such as a personal computer, server, smartphone, tablet computer, or industrial computer. The information processing device 400 includes, for example, a communication unit 401, an extraction unit 402, a projection unit 403, and a storage unit 410. The communication unit 401, extraction unit 402, and storage unit 410 can be modified using the same methods as the communication unit 101, extraction unit 102, and storage unit 110 described above, respectively, so their descriptions are omitted.

[0060] The projection unit 403 acquires motion data from the extraction unit 402. Based on the acquired motion data, the projection unit 403 creates projection information for projecting the person's movements into real space. The projection unit 403 transmits the created projection information from the communication unit 401 to the communication device 500. Regarding the projection information, the communication unit 401 receives either a first viewpoint switching request or a second viewpoint switching request transmitted by the communication device 500. The projection unit 403 obtains either the first viewpoint switching request or the second viewpoint switching request from the communication unit 401.

[0061] When the projection unit 403 receives a first viewpoint switching request, it creates first projection information for projecting the actions of a person as seen from the subjective first viewpoint when the first user U1 is acting independently, onto real space using a three-dimensional model. If the projection unit 403 receives a second viewpoint switching request after receiving a first viewpoint switching request, instead of projecting the actions of the person as seen from the subjective first viewpoint when the first user U1 is acting independently, onto the real space using the 3D model, it creates second projection information for projecting the actions of the person as seen from the second viewpoint when the actions of the first user U1's own 3D model are objectively observed onto the real space using the 3D model.

[0062] When the projection unit 403 receives a request to switch to a second viewpoint, it creates second projection information to project the actions of the person as seen from a second viewpoint—when objectively observing the actions of the 3D model of the first user U1—into real space using the 3D model. The projection unit 403 transmits either the created first projection information or the second projection information from the communication unit 401 to the communication device 500. If the projection unit 403 receives a request to switch to a first viewpoint after receiving a request to switch to a second viewpoint, instead of projecting the actions of the person as seen from the second viewpoint—when the actions of the first user U1's own 3D model are objectively observed—onto the real space using the 3D model, it creates first projection information for projecting the actions of the person as seen from the subjective first viewpoint—when the first user U1 is actively acting—onto the real space using the 3D model.

[0063] (Communication device 500) Figure 9 is a configuration diagram showing an example of a communication device 500 according to this embodiment. The communication device 500 includes, for example, a communication unit 510, a microphone 520, a detection device 530, an image display device 550, an HMI 560, and a control device 570. The detection device 530 includes, for example, a direction detection device 532, a head position detection device 534, and a motion sensor 536.

[0064] The communication unit 510, direction detection device 532, head position detection device 534, image display device 550, and HMI 560 can be replaced by the aforementioned communication unit 310, direction detection device 332, head position detection device 334, image display device 350, and HMI 360, respectively, so their descriptions are omitted.

[0065] Microphone 520 collects the voice emitted by the first user U1. The voice collected by microphone 520 is input to, for example, control device 570. The motion sensor 536 is a device for recognizing gesture operations performed by the first user U1. For example, a camera that captures images of the first user U1's upper body is used as the motion sensor 536. In this case, the control device 570 extracts characteristic points of the first user U1's body (fingertips, wrists, elbows, etc.) from the images captured by the camera and recognizes the first user U1's gesture operations based on the movement of these characteristic points.

[0066] The control device 570 includes, for example, a processor such as a CPU, and a storage medium connected to the processor that stores a program (set of instructions). The processor controls each part of the communication device 500 by executing the set of instructions. Note that all the functions of the communication device 500 shown in Figure 9 may be integrated into the VR goggles 555. The direction detection unit 572, head position detection unit 573, image editing unit 574, and image display control unit 575 can be replaced by the direction detection unit 372, head position detection unit 373, image editing unit 374, and image display control unit 375, respectively, so their descriptions are omitted.

[0067] The gesture input detection unit 576 detects the gesture input of the first user U1 based on the output of the motion sensor 536. The voice recognition unit 577 recognizes the voice of the first user U1 based on the output of the microphone 520.

[0068] The viewpoint switching request creation unit 578 creates a viewpoint switching request based on either or both of the gesture input of the first user U1 detected by the gesture input detection unit 576 and the voice of the first user U1 recognized by the voice recognition unit 577. Specifically, the viewpoint switching request creation unit 578 creates first projection request information when the gesture input of the first user U1 detected by the gesture input detection unit 576 indicates that the person (avatar) projected in real space is in the same or nearby position and in the same or similar posture. The viewpoint switching request creation unit 578 then creates a viewpoint switching request that includes the created first projection information.

[0069] The viewpoint switching request creation unit 578 creates second projection information when the voice of the first user U1 recognized by the voice recognition unit 577 indicates that the actions of a person as viewed from a second viewpoint are projected onto real space using a three-dimensional model. The viewpoint switching request creation unit 578 then creates a viewpoint switching request that includes the created second projection information. The viewpoint switching request creation unit 578 transmits the created viewpoint switching request from the communication unit 510 to the information processing device 400.

[0070] (Operation of the information processing system) Figure 10 is a flowchart showing an example of the operation of the information processing system 1 according to a modified example of the embodiment 1. (Step S1-2) The imaging unit 200 photographs and records people. Figure 11 is a diagram illustrating an example of the operation of the information processing system 1 according to a modified embodiment. Figure 11 shows an example of the imaging unit 200 photographing people. Figure 11 shows first user U1, second users U2-1 to U2-3 playing basketball. The imaging unit 200 includes imaging unit 200-1 and imaging unit 200-2. Imaging units 200-1 and 200-2 capture images of the first user U1, second user U2-1 through second user U2-3 playing basketball. Return to Figure 10 and continue the explanation. Steps S2-2 to S6-2 can be explained by applying steps S2-1 to S6-1 in Figure 7.

[0071] (Step S7-2) In the communication device 500, the gesture input detection unit 576 detects the gesture input of the first user U1 based on the output of the motion sensor 536, and determines whether or not it has detected a gesture input for viewpoint switching by determining whether or not the detected gesture input of the first user U1 indicates that the person (avatar) projected in real space is in the same or nearby position and in the same or similar posture.

[0072] (Step S8-2) In the communication device 500, if the gesture input detection unit 576 determines that the gesture input of the first user U1 detected is a viewpoint switching gesture input because the first user U1 is in the same or nearby position as the person (avatar) projected in real space and is in the same or similar posture, the viewpoint switching request creation unit 578 creates a first viewpoint switching request.

[0073] (Step S9-2) In the communication device 500, the communication unit 510 transmits the first viewpoint switching request created by the viewpoint switching request creation unit 578 to the information processing device 400. (Step S10-2) In the information processing device 400, the communication unit 401 receives a first viewpoint switching request from the communication device 500. The projection unit 403 acquires the first viewpoint switching request from the communication unit 401 and, based on the acquired first viewpoint switching request, creates first projection information for projecting the actions of a person as seen from the subjective first viewpoint when the first user U1 is acting independently, onto real space using a three-dimensional model. (Step S11-2) In the information processing device 400, the projection unit 403 transmits the created first projection information from the communication unit 401 to the communication device 500.

[0074] (Step S12-2) In the communication device 500, the communication unit 510 receives first projection information from the information processing device 400. The image editing unit 574 acquires the first projection information from the communication unit 510 and processes the acquired first projection information to create a projection image A1 that includes a three-dimensional model representing the actions of a person as seen from a subjective first viewpoint when the first user U1 is acting independently. The image editing unit 574 then extracts a projection image A2 corresponding to the direction of direction φ from the created projection image A1. The image editing unit 574 may also extract an image corresponding to the direction of direction for vertical movement of the head. The image display control unit 575 displays the projected image A2, which has been extracted by the image editing unit 574, on the image display device 550. This provides a mixed reality or augmented reality that represents the actions of a person as seen from the subjective first viewpoint when the first user U1 is acting independently, using a three-dimensional model. At this time, the image editing unit 574 may also display on the image display device 550 an image corresponding to the directional direction φ as seen from the height indicated by the height information of the first user U1's head. After that, the process proceeds to step S7-2.

[0075] (Step S13-2) If the communication device 500 does not determine that a viewpoint switching gesture input has been detected, the voice recognition unit 577 recognizes the voice of the first user U1 based on the output of the microphone 520, and determines whether or not the recognized voice of the first user U1 indicates that the actions of a person as seen from the second viewpoint are projected into real space using a 3D model, thereby determining whether or not the voice of viewpoint switching has been recognized. If it is determined that the voice of viewpoint switching has not been recognized, the process proceeds to step S6-2.

[0076] (Step S14-2) In the communication device 500, if the voice recognition unit 577 determines that it has recognized a voice for viewpoint switching because the voice of the recognized first user U1 indicates that the actions of a person as viewed from a second viewpoint are projected into real space using a three-dimensional model, the viewpoint switching request creation unit 578 creates a second viewpoint switching request. (Step S15-2) In the communication device 500, the communication unit 510 transmits the second viewpoint switching request created by the viewpoint switching request creation unit 578 to the information processing device 400.

[0077] (Step S16-2) In the information processing device 400, the communication unit 401 receives a second viewpoint switching request from the communication device 500. The projection unit 403 obtains the second viewpoint switching request from the communication unit 401 and, based on the obtained second viewpoint switching request, creates second projection information for projecting the actions of the person as seen from a second viewpoint—when objectively observing the actions of the 3D model of the first user U1 himself—into real space using the 3D model. (Step S17-2) In the information processing device 400, the projection unit 403 transmits the created second projection information from the communication unit 401 to the communication device 500.

[0078] (Step S18-2) In the communication device 500, the communication unit 510 receives second projection information from the information processing device 400. The image editing unit 574 acquires the second projection information from the communication unit 510 and processes the acquired second projection information to create a projection image A1 that includes a representation of the person's movements as seen from a second viewpoint when objectively observing the actions of the first user U1's own 3D model. The image editing unit 574 then extracts a projection image A2 corresponding to the directional direction φ from the created projection image A1. The image editing unit 574 may also extract an image corresponding to the directional direction for vertical movement of the head. The image display control unit 575 displays the projected image A2, which has been extracted by the image editing unit 574, on the image display device 550. This provides a mixed reality or augmented reality in which the actions of the person are represented by the 3D model as seen from a second viewpoint when objectively observing the actions of the 3D model of the first user U1. At this time, the image editing unit 574 may also display on the image display device 550 an image corresponding to the directional direction φ as seen from the height indicated by the height information of the first user U1's head. After that, the process proceeds to step S7-2.

[0079] Figure 12 is a diagram showing an example of the operation of an information processing system according to a modified embodiment. Figure 12(1) shows the 3D models Um2-1 to Um2-3 of the first user U1 and the second user. The actions of a person as seen from the subjective first viewpoint when the first user U1 is actively acting are projected into real space by the 3D models onto the VR goggles 555 worn by the first user U1. The 3D models projected into real space include the 3D models Um2-1 to Um2-3 of the second user.

[0080] Figure 12(2) shows the 3D model Um1 of the first user, and the 3D models Um2-1 through Um2-3 of the second user. The actions of the person as seen from a second viewpoint, when objectively observing the actions of the first user U1's own 3D model, are projected into real space by the 3D model onto the VR goggles 555 worn by the first user U1. The 3D models projected into real space include the 3D model Um1 of the first user, and the 3D models Um2-1 through Um2-3 of the second user.

[0081] When the first user U1 is at position (A), the VR goggles 555 display a 3D model of a person's movements as seen from a second viewpoint while U1 is at position (A) projected onto the real space. Similarly, when the first user U1 is at position (B), the VR goggles 555 display a 3D model of a person's movements as seen from a second viewpoint while U1 is at position (B) projected onto the real space. This allows the first user U1 to observe from any viewpoint.

[0082] In the modified embodiment 1 described above, the information processing device 100 may switch at any time between the subjective viewpoint of the first user U1 when acting proactively (hereinafter referred to as the "first viewpoint") and the viewpoint when objectively observing either or both of the actions of the first user U1 or the actions of other people other than the first user U1 (hereinafter referred to as the "second viewpoint"). After transmitting projection information to the communication device 500, the information processing device 400 receives either a first viewpoint switching request from the communication device 500, which requests to switch the viewpoint from the second viewpoint to the first viewpoint, or a second viewpoint switching request, which requests to switch the viewpoint from the first viewpoint to the second viewpoint.

[0083] Based on the received first viewpoint switching request, the information processing device 400 creates first projection information for projecting the actions of a person as seen from the subjective first viewpoint when the first user U1 is acting independently, onto real space using a three-dimensional model. Here, in the information processing device 400, the projection unit 403 may project the subject object (the three-dimensional model of the first user U1) transparently or semi-transparently so that it can be distinguished from the three-dimensional models of other people other than the first user U1. Furthermore, the projection unit 403 may perform scene control such as playback, reverse playback, and speed adjustment for the subject object. The information processing device 400 may, based on the received second viewpoint switching request, create second projection information for projecting the actions of a person as seen from a second viewpoint onto real space using a 3D model, when objectively observing either or both of the actions of the 3D model of the first user U1 or the actions of another person other than the first user U1.

[0084] In the first modified embodiment, the communication device 500 may be able to change the projection time when projecting a person's movements into real space. This allows, for example, if you want to check the person's movements again, you to project that part by rewinding the time. In the first modified embodiment, the first projection request information was created when the viewpoint switching request creation unit 578 creates first projection request information when the gesture input of the first user U1 detected by the gesture input detection unit 576 indicates that the first user U1 is in the same or nearby position as the person projected in real space and is in the same or similar posture. However, the invention is not limited to this example. For example, the operation that triggers the creation of the first projection request information can be set as appropriate.

[0085] In the first modified embodiment, the voice of the first user U1 recognized by the voice recognition unit 577 was used to describe the creation of second projection information when the voice indicates that the person's actions as viewed from a second viewpoint are projected into real space using a three-dimensional model. However, the invention is not limited to this example. For example, the operation that triggers the creation of the second projection request information can be set as appropriate. According to the modified embodiment 1, since the viewpoint can be switched between the subject and object in the real world, the first user U1 can observe their own actions from a free viewpoint, and since the projected time can be changed, it is possible to repeatedly practice the same scenario, which can contribute to improving one's own actions and learning.

[0086] Specifically, when the first user U1 assumes the same or a similar posture at the same or nearby position as the person (avatar) projected in real space, the gesture input detection unit 576 in the communication device 500 recognizes the position and posture of the first user U1. The viewpoint switching request creation unit 578 creates first projection request information based on the gesture input detection unit 576's recognition of the position and posture of the first user U1. In the information processing device 400, the projection unit 403 creates first projection information based on the first viewpoint switching request. In the communication device 500, the image editing unit 574 acquires the first projection information and processes the acquired first projection information to create a projection image A1 that includes a three-dimensional model representing the actions of a person as seen from the subjective first viewpoint when the first user U1 is acting independently. The image display control unit 575 displays the projection image A2 extracted by the image editing unit 574 on the image display device 550.

[0087] Subsequently, the first user U1 can control the playback scene by mimicking the actions of the person (avatar) projected into real space (their past actions). This allows the first user U1 to freely observe past scenarios from a subjective viewpoint, playing and rewinding them as needed. The playback speed may also be freely controlled.

[0088] (Modification 2) (Information Processing System) Figure 13 shows an example configuration of the information processing system 1 in a modified example of the embodiment 2. The information processing system 1 includes an information processing device 600, an imaging unit 200, and a communication device 300. The information processing device 600, the imaging unit 200, and the communication device 300 each communicate with each other via a network NW. The network NW includes, for example, at least one of the following: the Internet, WAN, LAN, mobile communication network, cellular network, etc.

[0089] Between the information processing device 600, the imaging unit 200, and the communication device 300, the communication device 300 displays the actions of a person created by the information processing device 600 based on at least a portion of the image of the person captured by the imaging unit 200, thereby providing the communication device 300 with augmented reality or mixed reality.

[0090] The information processing device 600 has the following functions in addition to the functions of the information processing device 100. The information processing device 600 projects the actions of multiple people into real space. As an example of multiple people, we will continue to explain the case in which oneself and others are projected into real space. Here, the others projected into real space may be one person or multiple people. The information processing device 600 has the function of extracting information useful for learning by analyzing the actions of oneself and others projected into real space. For example, the information processing device 600 compares one's own actions with the actions of others performing ideal actions. The information processing device 600 creates motion analysis information to present the results of comparing one's own actions with the actions of others, as well as points of focus and motion information when observing objects such as oneself and others. The information processing device 600 transmits the created motion analysis information, in addition to projection information for projecting a person's actions into real space, to the communication device 300.

[0091] The communication device 300 receives projection information and motion analysis information from the information processing device 600, and based on the received projection information and motion analysis information, projects its own and others' actions into real space, as well as projecting motion analysis information.

[0092] (Information processing device 600) The information processing device 600 is implemented by a device such as a personal computer, server, smartphone, tablet computer, or industrial computer. The information processing device 600 includes, for example, a communication unit 601, an extraction unit 602, a projection unit 603, an analysis unit 604, and a storage unit 610. The communication unit 601, extraction unit 602, and storage unit 610 can be modified using the same methods as the communication unit 101, extraction unit 102, and storage unit 110 described above, respectively, so their descriptions are omitted.

[0093] The projection unit 603 acquires motion data from the extraction unit 602. Based on the acquired motion data, the projection unit 603 creates projection information for projecting the person's movements into real space. The analysis unit 604 acquires projection information from the projection unit 603. Based on the acquired projection information, the analysis unit 604 compares its own movements with those of others. The analysis unit 604 creates motion analysis information to present the results of the comparison between its own movements and those of others, as well as points of focus and motion information when observing objects such as itself and others. The projection unit 603 acquires motion analysis information from the analysis unit 604 and transmits the acquired motion analysis information and the created projection information to the communication device 300 via the communication unit 601.

[0094] (Operation of the information processing system) Figure 14 is a flowchart showing an example of the operation of an information processing system according to a modified example 2 of the embodiment. Steps S1-3 to S4-3 can be explained by applying steps S1-1 to S4-1 in Figure 7.

[0095] (Step S5-3) In the information processing device 600, the analysis unit 604 acquires projection information from the projection unit 603 and compares its own movements with those of others based on the acquired projection information. The analysis unit 604 creates motion analysis information to present the results of the comparison between its own movements and those of others, as well as points of focus and motion information when observing objects such as itself and others. (Step S6-3) In the information processing device 600, the projection unit 603 acquires motion analysis information from the analysis unit 604, and transmits the acquired motion analysis information and the created projection information from the communication unit 601 to the communication device 300.

[0096] (Step S7-3) In the communication device 300, the communication unit 310 receives projection information and motion analysis information from the information processing device 600. The image editing unit 374 acquires the projection information and motion analysis information from the communication unit 310 and creates a projection image A1 including the motion analysis information by image processing the acquired projection information and motion analysis information. The image editing unit 374 then extracts a projection image A2 corresponding to the directional direction φ from the created projection image A1. The image display control unit 375 displays the projection image A2 extracted by the image editing unit 374 on the image display device 350.

[0097] Figure 15 is a diagram showing an example of the operation of an information processing system according to a modified example 2 of the embodiment. Figure 15 shows a 3D model Um1 of the first user and a 3D model Um2 of the second user. The actions of the first user U1 and the second user U2, as seen from the viewpoint of objectively observing the actions of the first user U1, are projected into real space by the 3D models onto the VR goggles 355 worn by the first user U1. The 3D models projected into real space include the 3D model Um1 of the first user and the 3D model Um2 of the second user.

[0098] The projected 3D model Um2 of the second user is performing ideal movements, and the 3D model Um1 of the first user is mimicking the movements of the second user's 3D model Um2. The VR goggles 355 worn by the first user U1 show a 3D model representation of the movements of the first user U1 and the second user U2 from the perspective of objectively observing the first user U1's own actions, and as an example of motion analysis information, it shows "Recall rate 45%" as the recall rate of the first user U1's movements for the second user U2, and attention information CI, which includes "position of the center of gravity, movement of the legs, and opening of the arms." The method for calculating the recall rate is predetermined. Furthermore, the analysis unit 604 may encourage users to focus on important features when projecting their own or others' actions into real space, using guidance such as light or sound, and may also project information about the actions obtained through motion analysis into space. This makes it possible for the first user U1 to acquire detailed information that cannot be obtained simply by observing the actions. Furthermore, the analysis unit 604 may be configured to reproduce the movements of the instructor, compare the user's own movements with ideal movements, and provide real-time suggestions for improvement. For example, if there are discrepancies in movement, the unit may automatically identify important times and movements that require particular attention and provide visual guidance or audio instructions for those points.

[0099] When the first user U1 is at position (A), the VR goggles 355 display a 3D model of a person's movements as seen from a second viewpoint when the first user U1 is at (A), projected onto real space. Similarly, when the first user U1 is at position (B), the VR goggles 355 display a 3D model of a person's movements as seen from a second viewpoint when the first user U1 is at (B), projected onto real space. Therefore, the first user U1 can observe from any viewpoint.

[0100] In a modified example of the embodiment 2, the communication device 300 may be able to change the projection time when projecting a person's movements into real space. This allows, for example, if you want to check the person's movements again, you to rewind time and project that part. According to the modified embodiment 2, it is possible to highlight aspects of one's own or others' behavior that should be focused on for learning purposes, thereby contributing to the improvement of one's own behavior and learning.

[0101] This section describes an example of the application of Information Processing System 1. (1) Observe one's own movements performed during sports training as an object. Traditionally, it was common to observe images captured by a camera on a 2D display, but by utilizing augmented reality and mixed reality technologies, it becomes possible to observe one's own movements in real space and from a free viewpoint, thereby promoting an understanding of one's own movements.

[0102] (2) The movements of the instructor performed during sports training are converted into a 3D model, and the participant performs the same movements while observing the model. At this time, the participant's own movements are analyzed in real time and compared with the ideal movements. After the movement is completed, the participant's own movements are converted into an object and compared with the 3D model of the ideal movement to understand the difference between themselves and the ideal. If there are areas for improvement, visual guidance and audio instructions are provided.

[0103] (3) By converting the group movements performed during sports training into a 3D model and displaying it in a reduced size, learners can be promoted to gain a spatial understanding from an overarching perspective. (4) By objectively reproducing the actions of multiple people in team sports or group activities (such as coordination in medical settings or within a company), the scenes in which mistakes occurred can be reviewed from an overview perspective, and instruction can be provided based on that review.

[0104] (5) If a mistake occurs while playing an instrument, observe the actions leading up to the mistake by objectively observing oneself. Then, by shifting the perspective from the moment immediately before the mistake to the subject, continue playing after the mistake and promote learning. (6) Review physical actions during customer service (bowing and gestures) and social behaviors such as bullying and violence as objectives, and encourage improvement and correction of one's own actions by taking the other person's perspective and the viewpoint of a third party. (7) Apply objectivity to training in settings where skills related to physical movement are required, such as medical settings.

[0105] Furthermore, it can be used as a new sports training system for team and individual sports, as a training system for skills-based activities such as playing musical instruments or operating machinery, and for educational purposes in schools and companies (reflection on physical movements and mental self-behavior).

[0106] According to the information processing system of the embodiment and modified embodiments 1 and 2, the information processing device includes an extraction unit that extracts motion data from a photographed image of a person including the user, and a projection unit that projects either or both of the user's motion and / or the motion of another person into real space based on the motion data extracted by the extraction unit. The projected motion of the user is objectively observable from the user's perspective. By configuring it in this way, the information processing device can project either the actions of a person, the actions of other people, or both, into real space based on motion data extracted from the captured image of the person. This allows for observation and learning of either the actions of a person, the actions of other people, or both, from a free viewpoint. This is effective in improving and learning behavior through free-viewpoint observation of one's own actions, the actions of others, or both. For example, it can be applied to supporting movement practice in sports, correcting behavior in customer service training, and correcting posture in private life. It is also effective in solving social problems by observing one's own actions from diverse viewpoints. For example, it can be applied to objectifying bullying and violence, and recreating scenes in which traffic accidents occur.

[0107] In an information processing device, the projection unit projects into real space at any desired timing. By configuring it in this way, the information processing device can project a person's actions into real space at any given time, allowing the perspectives of the subject and object to be switched at any point in time.

[0108] In the information processing device, when the projection unit projects a person's movements, it projects the projected movements of the person into real space along with pre-obtained desired movements of the person. By configuring it in this way, the information processing device can project the actions of the projected person into real space along with the desired actions of the person obtained in advance, thus allowing simultaneous observation of the projected person's actions and the desired actions of that person.

[0109] In an information processing device, the projection unit provides mixed reality or augmented reality. By configuring it in this way, the information processing device can reproduce human actions in the real world using augmented reality and mixed reality technologies. Because it can quickly reproduce similar actions in the real world using augmented reality and mixed reality technologies, it enables high-resolution and free-viewpoint observation that was not possible with conventional observation methods using 2D displays or VR.

[0110] In an information processing device, the projection unit, instead of projecting either one or both of a person's actions and / or another person's actions into real space, projects the actions of another person into real space as seen from the viewpoint of the person being projected into real space. By configuring it in this way, the information processing device can switch from projecting a person's actions onto either real space or the actions of other people, or both, to projecting the actions of other people, as seen from the viewpoint of the person projecting in real space, onto real space using a 3D model.

[0111] The information processing device includes a gesture input detection unit that detects a person's gesture input. The projection unit, based on the gesture input detected by the gesture input detection unit, projects the actions of other people as seen from the viewpoint of the person being projected into real space, instead of projecting either the person's actions or the actions of other people, or both, into real space. By configuring it in this way, the information processing device can switch from projecting either the actions of a person, the actions of another person, or both, into real space based on the person's gesture input, to projecting the actions of another person as seen from the viewpoint of the person currently projecting into real space. Therefore, a person can switch between projecting the actions of another person as seen from the viewpoint of the person currently projecting into real space without having to perform any operation on the information processing device.

[0112] In the information processing device, instead of projecting the actions of other people as seen from the viewpoint of the person being projected into real space, the projection unit projects either the actions of the person being projected into real space, or the actions of other people, or both, into real space based on action data extracted from the captured image of the person. By configuring it in this way, the information processing device can switch from projecting the actions of other people as seen from the viewpoint of a person projected into real space using a 3D model, to projecting either the actions of the person, the actions of other people, or both, into real space based on action data extracted from the captured image of the person.

[0113] In an information processing device, the projection unit projects either the actions of a person viewed from an arbitrary viewpoint, or the actions of another person, or both, into real space. By configuring it in this way, the information processing device can project either or both of the actions of a person viewed from an arbitrary viewpoint and the actions of other people into real space using a three-dimensional model.

[0114] In the information processing device, instead of projecting either or both of the actions of the person as seen from an arbitrary viewpoint and the actions of other people into real space, the projection unit projects the actions of the person into real space based on the action data extracted from the captured image of the person. By configuring it in this way, the information processing device can switch from projecting either the actions of a person or the actions of other people, or both, into real space based on motion data extracted from the captured image of the person, to projecting the actions of other people, as seen from the viewpoint of the person being projected into real space, using a 3D model.

[0115] The information processing device further includes a speech recognition unit that recognizes a person's voice. The projection unit switches from projecting the actions of other people as seen from the viewpoint of the person being projected into real space, based on the person's voice recognized by the speech recognition unit, to projecting either the actions of the person, the actions of other people, or both, into real space based on action data extracted from the captured image of the person. By configuring it in this way, the information processing device can switch from projecting the actions of other people in real space from the perspective of the person being projected in real space, based on the voice of the recognized person, to projecting either the actions of the person, the actions of other people, or both, into real space. Therefore, the person can switch between projecting either the actions of the person, the actions of other people, or both, into real space without having to perform any operation on the information processing device.

[0116] The information processing device further includes an analysis unit that compares the actions of a projected person with pre-obtained desired actions of the person. By configuring it in this way, the information processing device can compare the projected movements of the person with pre-obtained desired movements of the person, thereby acquiring detailed information that cannot be obtained simply by observing the movements.

[0117] In an information processing device, the imaging unit is one of the following: a motion capture camera, an RGB camera, an RGBD camera, or a spatial video camera, or a combination thereof. By configuring it in this way, the information processing device can acquire images of a person captured by one of the following: a motion capture camera, an RGB camera, an RGBD camera, a spatial video camera, or a combination thereof.

[0118] In an information processing device, the extraction unit converts motion data into either skeletal motion, mesh, or a 3D model. By configuring it in this way, the information processing device can acquire operational data.

[0119] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. For example, modified embodiment 1 and modified embodiment 2 may be combined. These embodiments and their modifications are included in the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as described in the claims.

[0120] Furthermore, the aforementioned information processing device 100, imaging unit 200, communication device 300, information processing device 400, communication device 500, and information processing device 600 each have a computer inside. The processes of each of the aforementioned devices are stored in program format on a computer-readable recording medium, and the above processes are performed when the computer reads and executes this program.

[0121] Here, computer-readable recording media refer to magnetic disks, magneto-optical disks, CD-ROMs, DVD-ROMs, semiconductor memory, etc. Alternatively, this computer program may be distributed to a computer via a communication line, and the computer that receives the program may execute it. Furthermore, the above program may be intended to implement some of the functions described above. Furthermore, the aforementioned functions may be realized in combination with programs already recorded in the computer device, such as so-called differential files (differential programs). [Explanation of symbols]

[0122] 100, 400, 600... Information processing unit, 101, 401, 601... Communication unit, 102, 402, 602... Extraction unit, 103, 403, 603... Projection unit, 110, 410, 610... Storage unit, 200... Imaging unit, 300, 500... Communication unit, 310, 510... Communication unit, 330, 530... Detection unit, 332, 532... Direction detection unit, 334, 534... Head position detection unit, 350, 550... Image display unit, 360, 560... HMI, 370, 570... Control unit, 372, 572... Direction detection unit, 373, 573... Head position detection unit, 374, 574... Image editing unit, 375, 575... Image display control unit, 520...Microphone, 536...Motion sensor, 576...Gesture input detection unit, ...Voice recognition unit, 578...Viewpoint switching request creation unit, 604...Analysis unit

Claims

1. A real-space projection method for projecting an object executed by a computer into real space, The steps of photographing people, including oneself, A step of extracting motion data from the image of the person that was photographed, The steps include projecting either the actions of the person in question or the actions of another person, or both, into real space based on the extracted motion data, It has, The actions of the projected person are objectively observable from the perspective of the self. A method of projection into real space.

2. In the aforementioned projection step, the projection is made into real space at any given time. The real space projection method according to claim 1.

3. In the projection step, if the person's actions are projected, the projected actions of the person are projected into the real space along with a pre-determined desired action of the person. The real space projection method according to claim 1.

4. The projection step provides mixed reality or augmented reality. The real space projection method according to claim 1.

5. Instead of projecting either or both of the aforementioned person's actions and the actions of the other person into real space, the step of projecting the actions of the other person as seen from the viewpoint of the aforementioned person projected into real space, It further possesses, The real space projection method according to claim 1.

6. A step of detecting the gesture input of the person, Based on the gesture input of the person detected in the detection step, instead of projecting either or both of the person's movements and / or the movements of the other person into real space, the step of projecting the movements of the other person as seen from the viewpoint of the person projected into real space is performed. It further possesses, The real space projection method according to claim 1.

7. Instead of projecting the actions of other people onto real space from the viewpoint of the person projected onto real space, the step involves projecting either the actions of the person, the actions of the other person, or both, onto real space based on the action data extracted from the captured image of the person. The real space projection method according to claim 5, further comprising the above.

8. In the projection step, the actions of the person and / or the actions of another person, as viewed from an arbitrary viewpoint, are projected into real space. The real space projection method according to claim 1.

9. Instead of projecting either or both of the person's movements from an arbitrary viewpoint and / or the movements of other people into real space, the step involves projecting the person's movements into real space based on the movement data extracted from the captured image of the person. The real space projection method according to claim 8, further comprising the above.

10. The steps include: recognizing the voice of the aforementioned person, Based on the voice of the person recognized in the recognition step, instead of projecting the actions of other people as seen from the viewpoint of the person projected into real space, the step of projecting either or both of the actions of the person and the actions of other people into real space based on the action data extracted from the captured image of the person, It further possesses, The real space projection method according to claim 1.

11. A step of comparing the projected movements of the person with the desired movements of the person obtained in advance. It further possesses, The real space projection method according to claim 3.

12. In the aforementioned shooting step, shooting is performed using one of the following: a motion capture camera, an RGB camera, an RGBD camera, or a spatial video camera, or a combination thereof. The real space projection method according to claim 1.

13. In the extraction step, the motion data is converted into either skeletal motion, mesh, or a 3D model. The real space projection method according to claim 1.

14. An extraction unit that extracts motion data from images of people, including oneself, taken by the camera, A projection unit projects either or both of the actions of the person and the actions of another person into real space based on the motion data extracted by the extraction unit. Equipped with, The actions of the projected person are objectively observable from the perspective of the self. Information processing device.

15. The imaging unit captures images of people, including the self, An extraction unit extracts motion data from an image of a person taken by the aforementioned imaging unit, A projection unit projects either or both of the actions of the person and the actions of other people into real space based on the motion data extracted by the extraction unit, Equipped with, The actions of the projected person are objectively observable from the perspective of the self. Information processing system.

16. On the computer, Steps include extracting motion data from images of people, including oneself, that have been photographed, A step of projecting either the actions of the person or the actions of another person, or both, onto real space based on the motion data extracted in the extraction step, Make it run, The actions of the projected person are objectively observable from the perspective of the self. program.

Images