Information processing device, information processing method, and program

Abstract

We propose an information processing device, information processing method, and program that can suppress the sudden interruption of the rendering of a body part even if the detection of that body part fails. [Solution] The first information processing device receives first information regarding the position and orientation of a first user. The first information processing device receives second information regarding a first physical object. In the first state, a first avatar corresponding to the first user is displayed based on the first information. In the first state, a virtual object corresponding to the first physical object is displayed based on the second information. The first information processing device receives first failure information indicating a failure to detect the first physical object. In the second state, the first avatar is displayed based on third information regarding the position and orientation of the first user. In the second state, the display of the virtual object is stopped based on the first failure information. The first physical object is an object held by the first user.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A technology for multiple users to share one virtual space using VR technology is known. Each user transmits the position and orientation information of a part of the body (for example, their own hand) recognized by a camera mounted on their own head-mounted display, together with the position and orientation information of the head, to other users. Each user generates an avatar image of other users based on the position and orientation information transmitted from other users.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When recognizing a user's hand using the camera image, the hand may go out of the camera's viewing angle and temporarily fail to be detected. In this case, the position and orientation information of the hand cannot be transmitted to other users. Therefore, on the other user side, it may seem that the hand has suddenly disappeared. This problem is not limited to the case of detecting the hand, but also applies to the case of detecting other parts of the body with a camera.

[0005] Therefore, the present disclosure proposes an information processing apparatus, an information processing method, and a program that can suppress the sudden interruption of the drawing of a part of the body even when the detection of a part of the body fails.

Means for Solving the Problems

[0006] According to this disclosure, an information processing method is provided that is performed by a computer, and includes: receiving first information relating to the position and orientation of a first user from a first information processing device; receiving second information relating to a first physical object from the first information processing device; displaying a first avatar corresponding to the first user based on the first information in a first state; displaying a virtual object corresponding to the first physical object based on the second information in the first state; receiving first failure information from the first information processing device indicating a failure to detect the first physical object; displaying the first avatar based on third information relating to the position and orientation of the first user in a second state; and stopping the display of the virtual object based on the first failure information in the second state, wherein the first physical object is an object held by the first user. Furthermore, according to this disclosure, a program is provided that causes a computer to implement the information processing of the information processing device. Furthermore, the present disclosure provides an information processing device comprising: a head information acquisition unit that acquires the position and posture of the user's head; a body information acquisition unit that acquires the position and posture of a part of the user's body other than the head; a determination unit that acquires the position and posture of a physical object grasped by the part of the body; and a communication unit that transmits information regarding the position and posture of the head, the position and posture of the part of the body, and the position and posture of the physical object to another information processing device, wherein the determination unit determines whether or not the part of the body is grasping the physical object, and if the determination unit determines that the part of the body is grasping the physical object and fails to detect the physical object, it transmits failure information indicating the failure to detect the physical object to the other information processing device, and does not transmit the failure information if the determination unit determines that the part of the body is not grasping the physical object and fails to detect the physical object. [Brief explanation of the drawing]

[0007] [Figure 1] This diagram explains the background of this disclosure. [Figure 2] This diagram explains the background of this disclosure. [Figure 3] This diagram explains the background of this disclosure. [Figure 4] This is a diagram showing an example of a terminal configuration. [Figure 5] This figure shows the transitions between detection states. [Figure 6] This flowchart shows an example of the process performed while tracking. [Figure 7] This figure shows an example of the processing flow during the recovery waiting period. [Figure 8] This diagram illustrates the process that occurs when transitioning from the tracking state to the recovery waiting state. [Figure 9] This figure shows an example of the transition process during the first transition period. [Figure 10] This figure shows an example of the hardware configuration of a terminal. [Figure 11] This figure shows an example of object detection processing for objects placed independently of the hand. [Figure 12] This figure shows an example of object detection processing when an object is being held in a hand. [Figure 13] This figure shows an example of a process for detecting whether an object is being held in a hand. [Figure 14] This figure shows an example of the processing flow during the waiting period when the hand is grasping an object. [Figure 15] This diagram shows an example of the processing flow during the waiting period for an object to be released when a hand is grasping it. [Figure 16] This diagram illustrates one example of a method for determining whether or not an object is being held in a hand. [Figure 17] This figure illustrates an example of information processing in the third embodiment. [Modes for carrying out the invention]

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] The description will be given in the following order. [1. First Embodiment] [1-1. Background] [1-2. Configuration of Terminal] [1-3. Transition of Detection State] [1-4. Information Processing Method] [1-5. Example of Hardware Configuration] [1-6. Effects] [2. Second Embodiment] [2-1. Information Processing Method] [2-2. Effects] [3. Third Embodiment] [3-1. Information Processing Method] [3-2. Effects]

[0010] [1. First Embodiment] [1-1. Background] FIGS. 1 to 3 are diagrams for explaining the background of the present disclosure.

[0011] As shown in FIG. 1, in the present disclosure, a case where a plurality of users U share one virtual space VSP via a terminal TM such as a head-mounted display (HMD) is assumed. Hereinafter, when distinguishing a plurality of users U, a number is attached after the symbol of the user U. When distinguishing a plurality of terminals TM, a number of the user U who holds the terminal TM is attached after the symbol of the terminal TM.

[0012] As shown in FIG. 2, the terminal TM has a display unit 60 and a sensor unit 10. The sensor unit 10 includes a camera CM. The camera CM has a visual field range FV in the direction that the user U views. The terminal TM detects an object OB in the real space RSP existing around the user U based on the video captured by the camera CM. The object OB includes a part of the body of the user U. In the example of FIG. 2, as a part of the body of the user U, a hand HN of the user U is detected. The terminal TM detects the position and posture of the hand HN and the position and posture of the head HD of the user U based on the sensor information. The terminal TM transmits information (position and posture information) regarding the positions and postures of the hand HD and the head HD to the terminal TM of another user U.

[0013] For example, in the example in Figure 2, user U1 is pointing with their hand HN. User U1's camera CM captures an image of the hand HN within the field of view FV. Terminal TM1 transmits information regarding the position and orientation of user U1's head HD and hand HN to terminal TM2. Terminal TM2 generates user U1's avatar AB based on the information transmitted from terminal TM1 and displays it on the display unit 60. Avatar AB includes a virtual object VOB corresponding to user U1's head HD and a virtual object VOB corresponding to the hand HN.

[0014] Multiple users U can communicate within the virtual space VSP via their avatars A and B. As shown in Figure 3, multiple users U are located in different places and cannot directly see each other. However, each user U can recognize the other users U by observing the movements of their avatars A and B displayed on the display unit 60. Although Figure 3 shows an example where multiple users U (user U1, user U2, and user U3) are located in different places in the real-world RSP (e.g., different rooms), multiple users U may be located in close proximity to each other (e.g., the same room).

[0015] Figure 1 shows user U1 attempting to explain a specific object POB to user U2 by pointing to it with their hand (HN). The top and middle sections of Figure 1 show the situation of each user U from the side and above, respectively. The bottom section of Figure 1 shows the image displayed on user U2's display unit 60.

[0016] The left side of Figure 1 shows user U1 guiding user U2 while looking in the direction indicated by hand HN. The right side of Figure 1 shows user U1 turning to explain to user U2. In the example on the left of Figure 1, user U1's hand HN is within the field of view FN of camera CM. Therefore, the display unit 60 of terminal TM2 displays the virtual object VOB corresponding to user U1's head HD and hand HN. In the example on the right of Figure 1, user U1's hand HN is outside the field of view FN of camera CM. Since the position and orientation of user U1's hand HN are not detected, only the position and orientation information of user U1's head HD is transmitted to terminal TM2. As a result, the display unit 60 of terminal TM2 displays only the virtual object VOB corresponding to user U1's head HD.

[0017] As described above, if the detection of the hand HN fails, the position and orientation information of the hand HN cannot be transmitted to other terminals TM. Other terminals TM will be unable to render the virtual object VOB corresponding to the hand HN, so the hand HN will appear to have temporarily disappeared.

[0018] User U typically doesn't pay attention to the camera's field of view (FV), so they are less likely to notice if their hand handle (HN) is outside the FV when conversing with another user U. Even if the hand handle is always within the FV, detection may temporarily fail due to lighting or other factors. If the hand handle that was previously displayed suddenly disappears, it may cause discomfort to user U2.

[0019] To keep the object to be recognized, OB, within the field of view (FV) at all times, it is necessary to either widen the field of view of the camera CM as much as possible or increase the number of camera CMs. Widening the field of view of the camera CM generally increases distortion of the input image, making correction more difficult and increasing the processing load. Increasing the number of camera CMs requires addressing increased costs, securing installation space, and increased power consumption and heat generation. Furthermore, it is necessary to accurately correlate the processing results from each camera CM in terms of time and space, making the processing more complex.

[0020] There are also examples of addressing the challenges of staff fighting without changing the field of view or number of camera CMs. Patent Document 1 describes a method to make the movement of avatar AB more natural by continuing the past state or transitioning to a predefined movement called "fluctuating movement" when the detection of the hand HN fails. However, if the past position is continuously used while the hand HN is moving, the movement of the hand HN will stop in place, which is thought to look unnatural. If a fluctuating movement is adopted, the disappearance of the virtual object VOB that existed up to that point can be avoided, but the connection to the previous movement is lost.

[0021] Therefore, this disclosure proposes a method for determining the current position and orientation of a hand HN based on previously detected hand HN position and orientation information. This method makes it less likely for the continuity of the hand HN position and orientation to be lost before and after a failure to detect the hand HN. Hereinafter, the position and orientation detected based on sensor information will be referred to as the actual position and orientation, and the position and orientation information related to the actual position and orientation will be referred to as the actual position and orientation information. Furthermore, the position and orientation estimated based on previously detected position and orientation information will be referred to as the provisional position and orientation, and the position and orientation information related to the provisional position and orientation will be referred to as the provisional position and orientation information.

[0022] [1-2. Terminal Configuration] Figure 4 shows an example of the configuration of terminal TM.

[0023] The terminal TM is an information processing device that processes various types of information based on sensor information. For example, the terminal TM includes a sensor unit 10, an information acquisition unit 20, a display control unit 30, a communication unit 50, a display unit 60, and a recognition result holding unit 70.

[0024] The sensor unit 10 includes various sensors for sensing the external environment. The sensor unit 10 includes camera CMs such as monocular cameras and stereo cameras. The sensor unit 10 can use different camera CMs depending on the application, such as for self-position estimation, object detection, and capturing images of the external environment for presentation to the user U. For example, the sensor unit 10 may include inertial sensors (accelerometers, gyroscopes), GPS (Global Positioning System), ultrasonic sensors, and distance measuring sensors to improve the accuracy of external environment recognition and reduce system latency.

[0025] The information acquisition unit 20 performs spatial recognition processing based on sensor information. This spatial recognition processing enables object detection and self-position estimation. Spatial recognition processing is performed for each frame of the camera CM.

[0026] Spatial recognition processing is performed using known techniques such as SLAM (Simultaneous Localization And Mapping). Spatial recognition processing may be performed using a general-purpose CPU (Central Processing Unit) or GPU (Graphics Processing Unit), or it may be performed using a processor specialized for image processing or machine learning.

[0027] The information acquisition unit 20 includes a head information acquisition unit 21 and a body information acquisition unit 22. The head information acquisition unit 21 acquires the actual position and orientation of the terminal TM based on self-position estimation. The actual position and orientation of the terminal TM represents the actual position and orientation of the head HD on which the terminal TM is attached. The body information acquisition unit 22 acquires the actual position and orientation of parts of the user U's body other than the head HD based on object detection. In this disclosure, the actual position and orientation of the user U's hand HD is acquired as part of the body, but the actual position and orientation of other parts of the user U, such as parts of the arm, may also be acquired. The information acquisition unit 20 sequentially outputs the actual position and orientation information of the user U's hand HN and head HD obtained by spatial recognition processing to the display control unit 30.

[0028] The display control unit 30 determines the position and orientation of the user U's hand HN and head HD, and controls the display of the display unit 60 based on the determination result. For example, during the tracking period when the body information acquisition unit 22 acquires the actual position and orientation of the hand HN, the display control unit 30 controls the display of the virtual object VOB corresponding to the hand HN based on the actual position and orientation of the hand HN and head HD. After the body information acquisition unit 22 temporarily fails to detect the hand HN, during the recovery waiting period until the body information acquisition unit 22 acquires the actual position and orientation of the hand HN again, the display control unit 30 controls the display of the virtual object VOB corresponding to the hand HN based on the history of the actual position and orientation of the hand HN and the actual position and orientation of the head HD. The history of the actual position and orientation refers to the time-series of actual position and orientation information acquired in the past, or the change in actual position and orientation over time as grasped based on this time-series of position information.

[0029] For example, the display control unit 30 includes a determination unit 31 and a drawing unit 32.

[0030] The determination unit 31 determines the position and orientation of the hand HN during the recovery waiting period when detection of the hand HN has failed, based on the history of the actual position and orientation of the hand HN immediately before detection of the hand HN began to fail. Unlike object detection, the actual position and orientation of the head HD is reliably acquired based on sensor information. The determination unit 31 determines that the actual position and orientation acquired by the head information acquisition unit 21 is the position and orientation of the head HD as is, and generates position and orientation information for the head HD. The determination unit 31 sequentially outputs the position and orientation information of the hand HN obtained by the determination, along with the actual position and orientation information of the head HD acquired from the head information acquisition unit 21, to the drawing unit 32 and the communication unit 50.

[0031] During the tracking period in which the hand HN has been successfully detected, the determination unit 31 determines the actual position and orientation of the hand HN and head HD as the position and orientation of the hand HN and head HD. The determination unit 31 sequentially outputs the actual position and orientation information of the hand HN and head HD acquired from the information acquisition unit 20 to the drawing unit 32 and the communication unit 50 as position and orientation information of the hand HN and head HD. The determination unit 31 also sequentially outputs the actual position and orientation information of the hand HN acquired from the body information acquisition unit 22 to the recognition result holding unit 70.

[0032] The recognition result holding unit 70 stores the actual position and orientation information of the hand HN output from the determination unit 31. The recognition result holding unit 70 receives the actual position and orientation information of the hand HN generated by the information acquisition unit 20 sequentially via the determination unit 31. Of the actual position and orientation information of the hand HN sequentially input from the determination unit 31, the recognition result holding unit 70 retains only the actual position and orientation information of the hand HN from the most recent multiple time points as the recognition result of the most recent hand HN. The recognition result holding unit 70 retains only the actual position and orientation information, and does not retain the provisional position and orientation information. Based on the actual position and orientation information of the hand HN held in the recognition result holding unit 70, the determination unit 31 determines the position and orientation of the hand HN during the recovery waiting period.

[0033] The communication unit 50 sequentially transmits the actual position and orientation information and the provisional position and orientation information output from the determination unit 31 to other terminals TM. The communication unit 50 outputs the actual position and orientation information and the provisional position and orientation information of other users U transmitted from other terminals TM to the drawing unit 32. The communication method may be wireless or wired.

[0034] The rendering unit 32 generates avatars A and B of the user U of the own aircraft based on the actual position and attitude information and the provisional position and attitude information input from the determination unit 31 of the own aircraft. The rendering unit 32 also generates avatars A and B of the user U of other terminals TM based on the actual position and attitude information and the provisional position and attitude information input from other terminals TM via the communication unit 50. A GPU is often used for rendering, but a CPU may also be used.

[0035] The display unit 60 displays the avatars A and B of the user U of the user U of the user U and other terminals TM, which are generated by the drawing unit 32. When the display unit 60 overlays the avatars A and B onto the real-space RSP, it also displays the video of the real-space RSP captured by the camera CM. The display unit 60 can be an LCD (Liquid Crystal Display) panel or an OLED (Organic Light-Rmitting Diode) panel, etc.

[0036] [1-3. Transition of detection state] Figure 5 shows the transitions in the detection state.

[0037] The IDLE state is when the information acquisition unit 20 is trying to detect the user's handle name but has not yet detected it. The tracking state is when the user's handle name was successfully detected during the previous attempt. The recovery waiting state is when the user's handle name has failed to be detected, and the pre-set time limit has not been exceeded since the failure began.

[0038] The IDLE state includes a long-term error state where the time elapsed since the start of failures in detecting the hand hand number exceeds the time limit, and an unestimable state where the position and orientation of the hand hand number cannot be properly estimated based on the history of the hand hand number's actual position and orientation. The unestimable state includes a state where the estimation results have broken down, and a state where the position and orientation of the hand hand number cannot be properly estimated because past changes in the actual position and orientation, as determined from the history of the hand hand number's actual position and orientation, exceed the acceptable range. Both the long-term error state and the recovery waiting state are states where detection failed during the previous detection attempt, but they are distinguished by the elapsed time since the start of detection failures.

[0039] In this disclosure, the periods during which the IDLE state persists, the periods during which the tracking state persists, and the periods during which the recovery waiting state persists are referred to as the IDLE period, the tracking period, and the recovery waiting period, respectively.

[0040] During the IDLE period, it is highly likely that the appropriate position and orientation of the hand HN cannot be estimated based on the history of the hand HN's actual position and orientation. For example, in a long-term error state, hand HN detection has continuously failed for a long time, and the position and orientation of the hand HN may have changed significantly beyond expectations. In an unestimable state, appropriate estimation cannot be performed, or the estimation results may break down. Therefore, during the IDLE period, the determination unit 31 does not estimate the position and orientation of the hand HN. The determination unit 31 stops estimating the position and orientation of the hand HN until it succeeds in detecting the hand HN again. The determination unit 31 either does not send the hand HN position and orientation information to other terminals TM, or sends default hand HN position and orientation information to other terminals TM. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until it succeeds in detecting the hand HN again.

[0041] During the IDLE period, the determination unit 31 may send information (failure information) to other terminals TM indicating that it has failed to detect the hand name. When other terminals TM receive the failure information, they can notify the user U of the failure information via a pop-up display or other means.

[0042] During the tracking period, the information acquisition unit 20 generates actual position and orientation information of the hand HN. The determination unit 31 determines the actual position and orientation of the hand HN as the position and orientation of the hand HN and transmits the actual position and orientation information of the hand HN to other terminals TM. The drawing unit 32 controls the display of the virtual object VOB corresponding to the hand HN based on the actual position and orientation of the hand HN.

[0043] During the recovery waiting period, the determination unit 31 estimates the position and orientation of the hand HN based on the history of the hand HN's actual position and orientation immediately before it began to fail to detect the hand HN, and the actual position and orientation of the head HD. Based on the estimated position and orientation of the hand HN, the determination unit 31 generates provisional position and orientation information and transmits it to other terminals TM. Based on the provisional position and orientation of the hand HN, the determination unit 31 controls the display of the virtual object VOB corresponding to the hand HN.

[0044] The determination unit 31 may adjust the length of the time limit according to the magnitude of past changes in the actual position and posture of the hand HN, which are detected based on the history of the actual position and posture of the hand HN. For example, the determination unit 31 detects the magnitude of the change in the actual position and posture of the hand HN immediately before detection of the hand HN begins to fail, based on the actual position and posture information of the hand HN at multiple time points held in the recognition result holding unit 70, as the magnitude of past changes in the actual position and posture of the hand HN. The determination unit 31 sets a longer time limit the larger the change in the actual position and posture of the hand HN. However, if the detection failure is temporary and is not expected to continue for a long time, such as when the detection failure is not due to movement of the field of view FV but mainly due to image processing problems (e.g., the effect of light), the time limit does not need to be adjusted.

[0045] [1-4. Information Processing Methods] An example of information processing in the determination unit 31 will be explained below using Figures 6 and 7.

[0046] Figure 6 is a flowchart showing an example of the process performed in tracking mode.

[0047] In step S1, the determination unit 31 determines whether the information acquisition unit 20 has successfully detected the hand HN. For example, if the determination unit 31 has acquired the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has been successful. If the determination unit 31 has not acquired the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has failed.

[0048] If it is determined in step S1 that the detection of the hand HN was successful (step S1: Yes), the process proceeds to step S2. In step S2, the determination unit 31 determines that the actual position and orientation of the hand HN obtained from the information acquisition unit 20 is the current position and orientation of the hand HN. The determination unit 31 outputs the actual position and orientation information of the hand HN to the drawing unit 32, the communication unit 50, and the recognition result holding unit 70. The drawing unit 32 controls the display of the virtual object OB corresponding to the hand HN based on the determined position and orientation of the hand HN. Subsequently, in step S3, the determination unit 31 performs a termination determination and repeats the above process until a termination flag, such as a termination operation by user U, is detected.

[0049] If it is determined in step S1 that the detection of the hand handle has failed (step S1: No), the determination unit 31 performs the recovery waiting process flow described later.

[0050] Figure 7 shows an example of the processing flow during the recovery waiting period. Figure 8 is a diagram illustrating the processing that takes place when transitioning from the tracking state to the recovery waiting state.

[0051] In step S10, the determination unit 31 determines whether the change in the actual position and orientation of the hand HN immediately before the detection of the hand HN begins to fail is within an acceptable range.

[0052] For example, the determination unit 31 obtains actual position and orientation information of the hand HN at multiple time points immediately before the detection of the hand HN begins to fail from the recognition result holding unit 70. As shown in Figure 8, the determination unit 31 calculates the movement speed and orientation change speed of the hand HN immediately before the detection of the hand HN begins to fail, based on the actual position and orientation PSA of the hand HN at multiple time points. If the movement speed and orientation change speed of the hand HN meet a preset criterion, the determination unit 31 determines that the change in the actual position and orientation PSA is within an acceptable range. In the example in Figure 8, the actual position and orientation PSAs PSA1 to PSA3 at three time points are shown in chronological order, but the number of actual position and orientation PSAs to be calculated is not limited to three.

[0053] In step S10, if it is determined that the change in the actual position and orientation PSA exceeds the acceptable range (step S10: No), the detection state of the hand HN transitions to the IDLE state. The determination unit 31 does not estimate the position and orientation of the hand HN. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until the hand HN is successfully detected.

[0054] In step S10, if it is determined that the change in the actual positional posture PSA is within an acceptable range (step S10: Yes), the process proceeds to step S11. In step S11, the determination unit 31 estimates the current position and posture of the hand HN based on the history of the most recent actual positional posture PSA of the hand HN. For example, as shown in Figure 8, the determination unit 31 calculates the current position and posture of the hand HN as a provisional positional posture PSE, assuming that the position and posture of the hand HN have changed up to the present at the rate of movement and posture change of the hand HN determined in step S10. The determination unit 31 estimates the provisional positional posture PSE of the hand HN obtained by the calculation as the current position and posture of the hand HN.

[0055] In step S12, the determination unit 31 estimates the joint movements of user U from the provisional position and posture PSE of hand HN based on IK (inverse kinematics). The determination unit 31 determines whether the estimated joint movements satisfy the joint constraints.

[0056] If it is determined in step S12 that the joint movement satisfies the joint constraints (step S12: Yes), the process proceeds to step S13. In step S13, the determination unit 31 outputs temporary position and posture information regarding the temporary position and posture PSE to the drawing unit 32 and the communication unit 50. If it is determined in step S12 that the joint movement does not satisfy the joint constraints (step S12: No), the detection state of the hand HN transitions to the IDLE state. The determination unit 31 rejects the estimation results of the hand HN's position and posture and stops estimating the hand HN's position and posture until it succeeds in detecting the hand HN. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until it succeeds in detecting the hand HN.

[0057] Inverse Kinematics (IK) is a common technique that determines the position and orientation of a joint between specified start, end, and target points for a given group of joints through interpolation calculations. As shown in Figure 8, the hand human nose (HN) is connected to the head human body (HD) via joints. Therefore, constraints based on the structure of the human body exist between the hand HN and the head HD.

[0058] For example, the determination unit 31 sets the actual position and posture PSA of the hand HN detected by the body information acquisition unit 22 in the previous instance as the endpoint, the actual position and posture of the head HD detected by the head information acquisition unit 21 at the same time as the previous instance as the starting point, and the provisional position and posture PSE of the hand HN estimated by the determination unit 31 in the current instance as the target point. If, during the process of moving from the endpoint to the target point, the predetermined constraints of the rotation angles of each joint are exceeded, or the maximum distance between the starting point and the endpoint is exceeded, the determination unit 31 determines that the relationship between the provisional position and posture PSE of the hand HN and the actual position and posture of the head HD detected by the information acquisition unit 20 is broken, and rejects the estimation result. If the estimation result is not rejected, the determination unit 31 determines that the provisional position and posture PSE is valid. The determination unit 31 determines that the provisional position and posture PSE is the current position and posture of the hand HN and notifies the drawing unit 32 and the communication unit 50.

[0059] If the provisional position and orientation information is output to the drawing unit 32 and the communication unit 50 in step S13, then in step S14, the determination unit 31 determines whether or not a hand HN has been detected. For example, if the determination unit 31 has obtained the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has been successful. If the determination unit 31 has not obtained the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has failed.

[0060] If it is determined in step S14 that the detection of the hand handle was successful (step S14: Yes), the process proceeds to step S15. In step S15, the determination unit 31 performs a process to transition to the normal tracking process.

[0061] If the detection of the hand HN is successful, the detection state of the hand HN transitions to the tracking state. As shown in step S2, in the tracking state, the actual position and attitude PSA detected based on the sensor information is used as the position and attitude of the hand HN. However, if the actual position and attitude PSA is used as is, a large discrepancy may occur between it and the previous temporary position and attitude PSE due to position and attitude errors accumulated during the recovery waiting period. Therefore, the determination unit 31 sets a predetermined period immediately after the hand HN detection has resumed as the first transition period, and gradually reduces the aforementioned discrepancy within the first transition period.

[0062] For example, during the first transition period, the determination unit 31 determines the current position and orientation of the hand HN by determining the intermediate position orientation PSM (see Figure 9) between the current actual position orientation PSA of the hand HN based on sensor information acquired by the body information acquisition unit 22 and the provisional position orientation PSE of the hand HN estimated based on the history of past actual position orientation PSAs of the hand HN. The determination unit 31 outputs the provisional position orientation information of the hand HN obtained by the determination to the drawing unit 32 and the communication unit 50. The drawing unit 32 controls the display of the virtual object VOB corresponding to the hand HN based on the determined intermediate position orientation PSM.

[0063] In step S16, the determination unit 31 determines whether the first transition period has elapsed. If it is determined in step S16 that the first transition period has elapsed (step S16: Yes), the process proceeds to step S1, and the determination unit 31 performs the normal tracking state processing. After the first transition period has elapsed, until the detection of object OB fails again, the determination unit 31 determines the actual position and orientation PSA of the hand HN detected based on the sensor information as the position and orientation of the hand HN.

[0064] If it is determined in step S16 that the first transition period has not elapsed (step S16: No), the process returns to step S14 and continues the transition process from steps S14 to S16 until the first transition period ends.

[0065] Figure 9 shows an example of the transition process during the first transition period.

[0066] Time t1 indicates the time when the detection of object OB was restored. The determination unit 31 sets the period from time t1 to time t3 as the first transition period.

[0067] At time t1, the determination unit 31 estimates the provisional position and posture PSE of the hand HN at time t1 based on the history of the hand HN's past actual position and posture. For example, the determination unit 31 calculates the position and posture of the hand HN at time t1 as the provisional position and posture PSE at time t1, assuming that the position and posture of the hand HN changed up to time t1 at the speed of movement and posture change of the hand HN determined in step S4. The determination unit 31 acquires the actual position and posture information at time t1, generated from the information acquisition unit 20 based on the sensor information at time t1. The determination unit 31 determines that the position and posture of the hand HN at time t1 is the intermediate position and posture PSM1 between the actual position and posture PSA at time t1, indicated by the actual position and posture information at time t1, and the provisional position and posture PSE at time t1, calculated based on the history of past actual position and posture.

[0068] At time t2, the determination unit 31 acquires actual position and orientation information for time t2, generated from the information acquisition unit 20 based on the sensor information for time t2. The determination unit 31 determines that the intermediate position orientation PSM2 between the actual position and orientation PSA at time t2, indicated by the actual position and orientation information for time t2, and the intermediate position and orientation PSM1 at time t1, obtained by the determination at time t1, is the position and orientation of the hand HN at time t2.

[0069] At time t3, the determination unit 31 acquires actual position and orientation information for time t3, generated from the information acquisition unit 20 based on the sensor information for time t3. The determination unit 31 determines that the intermediate position orientation PSM3 between the actual position and orientation PSA at time t3, indicated by the actual position and orientation information for time t3, and the intermediate position and orientation PSM2 at time t2, obtained by the determination at time t2, is the position and orientation of the hand HN at time t3.

[0070] From time t4 onward, the determination unit 31 uses the actual position and attitude PSA indicated by the actual position and attitude information as the position and attitude of the hand HN.

[0071] Return to Figure 7. If it is determined in step S14 that the detection of the hand handle has failed (step S14: No), proceed to step S17. In step S17, the determination unit 31 determines whether the time during which the failure to detect the hand handle continues has exceeded a predetermined time limit.

[0072] In step S17, if it is determined that the time during which the failure continues exceeds the time limit (step S17: Yes), the process proceeds to step S18. In step S18, the determination unit 31 notifies the drawing unit 32 and the communication unit 50 that detection of the hand HN has failed. The detection state of the hand HN transitions to the IDLE state. The determination unit 31 stops determining the position and orientation of the hand HN until detection of the hand HN is successful. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until detection of the hand HN is successful. In step S17, if it is determined that the time limit has not elapsed (step S17: No), the process returns to step S11.

[0073] [1-5. Hardware Configuration Examples] Figure 10 shows an example of the hardware configuration of a terminal TM.

[0074] The terminal TM has a computer 1000 as shown in Figure 10. The computer 1000 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The computer 1000 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. Furthermore, the computer 1000 may optionally include an imaging device 933 and a sensor 935. The computer 1000 may have a processing circuit called a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of, or together with, the CPU 901.

[0075] The CPU 901 functions as an arithmetic processing unit and control unit, controlling all or part of the operation within the computer 1000 according to various programs recorded in the ROM 903, RAM 905, storage device 919, or removable recording medium 927. The ROM 903 stores programs and arithmetic parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. The CPU 901, ROM 903, and RAM 905 are interconnected by a host bus 907, which is composed of internal buses such as the CPU bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0076] The input device 915 is a device operated by the user, such as a mouse, keyboard, touch panel, buttons, switches, and levers. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may also be a remote control device that uses infrared or other radio waves, or an external connection device 929 such as a mobile phone that is compatible with the operation of the computer 1000. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the CPU 901. By operating this input device 915, the user inputs various data to the computer 1000 or instructs it to perform processing operations. The imaging device 933, which will be described later, can also function as an input device by imaging the user's hand movements, fingers, etc. In this case, the pointing position may be determined according to the hand movements and finger orientation.

[0077] The output device 917 is comprised of a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may include, for example, display devices such as LCDs (Liquid Crystal Displays), PDPs (Plasma Display Panels), OLED (Electro-Luminescence) displays, and projectors; hologram display devices; audio output devices such as speakers and headphones; and printers. The output device 917 outputs the results obtained from the processing of the computer 1000 as text or images, or as audio such as voice or sound. The output device 917 may also include a light to illuminate the surroundings.

[0078] The storage device 919 is a data storage device configured as an example of the memory unit of the computer 1000. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs executed by the CPU 901, various data, and various data acquired from external sources.

[0079] Drive 921 is a reader / writer for removable recording media 927, such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the computer 1000. Drive 921 reads information recorded on the installed removable recording media 927 and outputs it to RAM 905. Drive 921 also writes data to the installed removable recording media 927.

[0080] Connection port 923 is a port for directly connecting equipment to computer 1000. Connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, connection port 923 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting external equipment 929 to connection port 923, various types of data can be exchanged between computer 1000 and external equipment 929.

[0081] The communication device 925 is a communication interface, for example, consisting of a communication device for connecting to the communication network 931. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). Alternatively, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. The communication device 925 sends and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The communication network 931 connected to the communication device 925 is a network connected by wire or wireless, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0082] The imaging device 933 is a device that captures real space and generates an image using various components such as an image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and a lens for controlling the imaging of a subject onto the image sensor. The imaging device 933 may capture still images or it may capture moving images.

[0083] Sensor 935 is a variety of sensors, such as a distance sensor, acceleration sensor, gyroscope, geomagnetic sensor, light sensor, and sound sensor. Sensor 935 acquires information about the state of the computer 1000 itself, such as the orientation of the computer 1000's casing, and information about the surrounding environment of the computer 1000, such as the brightness and noise level around the computer 1000. Sensor 935 may also include a GPS sensor that receives GPS (Global Positioning System) signals and measures the latitude, longitude, and altitude of the device.

[0084] [1-6. Effects] The terminal TM includes a head information acquisition unit 21, a body information acquisition unit 22, and a display control unit 30. The head information acquisition unit 21 acquires the position and orientation of the user U's head HD. The body information acquisition unit 22 acquires the position and orientation of the user U's hand HN. The display control unit 30 controls the display of the virtual object VOB corresponding to the hand HN during the recovery waiting period from the failure to detect the hand HN until the body information acquisition unit 22 acquires the position and orientation of the hand HN, based on the history of the actual position and orientation of the hand HN and the actual position and orientation of the head HD. In this embodiment, the information processing method of the terminal TM described above is executed by the computer 1000. The program of this embodiment causes the computer 1000 to implement the terminal TM processing described above.

[0085] This configuration prevents sudden interruptions in the rendering of the user's handle name (HN) even if detection fails.

[0086] The display control unit 30 estimates the position and orientation of the hand HN based on the history of the hand HN's actual position and orientation and the head HD's actual position and orientation, and controls the display of the virtual object VOB corresponding to the hand HN based on the estimation result. The display control unit 30 detects changes in the hand HN's actual position and orientation based on the history of the hand HN's actual position and orientation. If the detected change in the hand HN's actual position and orientation exceeds an acceptable range, the display control unit 30 does not estimate the position and orientation of the hand HN and stops the display of the virtual object VOB corresponding to the hand HN.

[0087] This configuration suppresses the occurrence of judgments that deviate significantly from the actual position and orientation of the hand HN. It is difficult to accurately predict the position and orientation of a hand HN that is constantly changing. Forcing a position and orientation determination in such cases can lead to unnatural results. By stopping the determination process, such unnaturalness becomes less likely.

[0088] The display control unit 30 estimates the joint movements of the user U based on IK, from the estimated position and orientation of the hand HN. If the estimated joint movements do not satisfy the joint constraints, the display control unit 30 rejects the estimation results of the position and orientation of the hand HN and stops displaying the virtual object VOB corresponding to the hand HN.

[0089] This configuration makes it less likely for unnatural judgments that violate the constraints of the joints to occur.

[0090] The display control unit 30 stops displaying the virtual object VOB corresponding to the hand HN if the recovery waiting period until the actual position and orientation of the hand HN is acquired by the body information acquisition unit 22 exceeds a preset time limit.

[0091] This configuration helps to suppress the accumulation of errors in the judgment process, which can impair the validity of the judgment result.

[0092] The display control unit 30 adjusts the length of the time limit according to the magnitude of past changes in the actual position and orientation of the hand HN, which are detected based on the history of the actual position and orientation of the hand HN.

[0093] This configuration allows the time limit to be flexibly adjusted in response to changes in position and orientation just before detection begins to fail. As a result, the validity of the position and orientation determination results during the recovery waiting period is increased.

[0094] When the display control unit 30 transitions from a state where hand HN detection has failed to a state where it has succeeded, it determines the position and orientation of the hand HN during the first transition period immediately after the hand HN detection returns due to the transition as an intermediate position orientation PSM between the actual position orientation PSA of the hand HN acquired by the body information acquisition unit 22 and the provisional position orientation PSE of the hand HN estimated based on the history of the actual position orientation of the hand HN, and controls the display of the virtual object VOB corresponding to the hand HN based on the determined position and orientation.

[0095] This configuration allows for the gradual reduction of positional and orientation errors accumulated during the recovery waiting period within the first transition period.

[0096] The display control unit 30 determines the position and orientation of the hand HN after the first transition period has elapsed as the actual position and orientation PSA acquired by the body information acquisition unit 22, and controls the display of the virtual object VOB corresponding to the hand HN based on the determined position and orientation.

[0097] With this configuration, the position and orientation PS during the first transition period and the position and orientation PS after the end of the first transition period change smoothly. Therefore, the continuity of position and orientation after recovery from failure is enhanced.

[0098] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0099] [2. Second Embodiment] [2-1. Information Processing Methods] Figures 11 to 15 show an example of information processing according to the second embodiment.

[0100] The difference between this embodiment and the first embodiment is that the determination method of this disclosure is applied to the object OB held by the hand HN. When the hand HN is grasping the object OB, the display control unit 30 controls the display of the virtual object VOB corresponding to the object OB during the recovery waiting period until the position and orientation of the object OB are acquired, based on the history of the position and orientation of the object OB. When the hand HN is not grasping the object OB, the display control unit 30 stops the display of the virtual object VOB corresponding to the object OB. The differences from the first embodiment will be described below.

[0101] Figure 11 shows an example of the detection process for object OB, which is positioned independently of hand HN.

[0102] In step S21, the determination unit 31 determines whether the information acquisition unit 20 has successfully detected object OB. If it is determined in step S21 that object OB has been successfully detected (step S21: Yes), the process proceeds to step S22. In step S22, the determination unit 31 outputs the actual position and orientation information of object OB acquired from the information acquisition unit 20 to the drawing unit 32, the communication unit 50, and the recognition result holding unit 70. Subsequently, in step S23, the determination unit 31 performs a termination determination and repeats the above process until a termination flag is detected.

[0103] If it is determined in step S21 that object OB detection has failed (step S21: No), step S21 is repeated until object OB is successfully detected. Unlike the hand HN, object OB has no distance constraints from the user U. Therefore, even if the position and orientation at the time of detection failure are determined as with the hand HN, the validity of the determination result cannot be judged based on IK. For this reason, if object OB detection fails, the detection state of object OB transitions to the IDLE state.

[0104] Figure 12 shows an example of the object detection process when object OB is being held by hand HN.

[0105] As mentioned above, if object OB is not detected, the detection state of object OB does not transition to the recovery waiting state, but to the IDLE state. However, if object OB is being held by hand HN and object OB is not detected, the detection state of object OB transitions to the recovery waiting state. If both the detection state of object OB and hand HHN are in the tracking state, and the distance between object OB and hand HN is smaller than a preset threshold, the determination unit 31 determines that object OB is being held by hand HN.

[0106] Figure 16 illustrates an example of a method for determining whether or not object OB is being held by hand HN.

[0107] Whether or not object OB is being grasped by hand HN is determined by the relationship between the detected object OB's actual position and orientation (zObj) and three of the detected hand HN's major joints (zJ1, zJ2, zJ3). An example of this calculation is shown below. In the formula below, "Abs" represents the absolute value, and "threshold" represents the criterion for determining whether or not it is being grasped.

[0108]

number

[0109] Figure 13 shows an example of the detection process for when object OB is being held by hand HN.

[0110] In step S31, the determination unit 31 determines whether the information acquisition unit 20 has successfully detected the hand HN. For example, if the determination unit 31 has acquired the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has been successful. If the determination unit 31 has not acquired the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has failed.

[0111] If it is determined in step S31 that the detection of the hand HN was successful (step S31: Yes), the process proceeds to step S32. In step S32, the determination unit 31 determines whether the information acquisition unit 20 was successful in detecting object OB. For example, if the determination unit 31 obtains the actual position and orientation information of object OB from the information acquisition unit 20, it determines that the detection of object OB was successful. If the determination unit 31 does not obtain the actual position and orientation information of object OB from the information acquisition unit 20, it determines that the detection of object OB was unsuccessful.

[0112] In step S32, if it is determined that object OB has been successfully detected (step S32: Yes), the process proceeds to step S33. In step S33, the determination unit 31 outputs the actual position and orientation information of object OB and hand HN obtained from the information acquisition unit 20 to the drawing unit 32, the communication unit 50, and the recognition result holding unit 70. Then, in step S34, the determination unit 31 performs a termination determination and repeats the above process until a termination flag is detected.

[0113] If it is determined in step S31 that the detection of the hand handle has failed (step S31: No), the determination unit 31 performs the recovery waiting process flow described later.

[0114] If it is determined in step S32 that object OB detection has failed (step S32: No), the object OB detection state transitions to a recovery waiting state. The determination unit 31 then performs the processing flow for waiting for object OB to recover.

[0115] Figure 14 shows an example of the processing flow during the waiting period for hand HN to return when hand HN is gripping object OB.

[0116] In step S40, the determination unit 31 determines whether the change in the actual position and orientation of the hand HN immediately before the detection of the hand HN begins to fail is within an acceptable range.

[0117] In step S40, if it is determined that the change in the position and orientation of the hand HN exceeds the acceptable range (step S40: No), the detection state of the hand HN transitions to the IDLE state. The determination unit 31 does not estimate the position and orientation of the hand HN. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until the hand HN is successfully detected.

[0118] If, in step S40, it is determined that the change in the actual position and orientation of the hand HN is within an acceptable range (step S40: Yes), the process proceeds to step S41. In step S41, the determination unit 31 estimates the current position and orientation of the hand HN based on the history of the most recent actual position and orientation PSA of the hand HN.

[0119] In step S42, the determination unit 31 estimates the joint movements of user U based on IK from the provisional position and posture PSE of hand HN. The determination unit 31 determines whether the estimated joint movements satisfy the joint constraints.

[0120] If it is determined in step S42 that the joint movement satisfies the joint constraints (step S42: Yes), the process proceeds to step S43. In step S43, the determination unit 31 outputs temporary position and posture information regarding the temporary position and posture PSE to the drawing unit 32 and the communication unit 50. If it is determined in step S42 that the joint movement does not satisfy the joint constraints (step S42: No), the detection state of the hand HN transitions to the IDLE state. The determination unit 31 rejects the estimation results of the position and posture of the hand HN and stops estimating the position and posture of the hand HN until it succeeds in detecting the hand HN. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until it succeeds in detecting the hand HN.

[0121] If the provisional position and orientation information is output to the drawing unit 32 and the communication unit 50 in step S43, then in step S44, the determination unit 31 determines whether or not a hand HN has been detected. For example, if the determination unit 31 has obtained the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has been successful. If the determination unit 31 has not obtained the actual position and orientation information of the hand HN from the information acquisition unit 20, it determines that the detection of the hand HN has failed.

[0122] If it is determined in step S44 that the detection of the hand handle was successful (step S44: Yes), the process proceeds to step S45. In step S45, the determination unit 31 performs a process to transition to the normal tracking process.

[0123] If the detection of the hand HN is successful, the detection state of the hand HN transitions to the tracking state. As shown in step S33, in the tracking state, the actual position and attitude PSA detected based on the sensor information is used as the position and attitude of the hand HN. However, if the actual position and attitude PSA is used as is, a large discrepancy may occur between it and the previous temporary position and attitude PSE due to position and attitude errors accumulated during the recovery waiting period. Therefore, the determination unit 31 sets a predetermined period immediately after the hand HN detection has returned as the first transition period, and gradually reduces the aforementioned discrepancy within the first transition period.

[0124] For example, during the first transition period, the determination unit 31 determines that the current position and orientation of the hand HN is the intermediate position orientation PSM between the current actual position orientation PSA based on sensor information acquired by the body information acquisition unit 22 and the current provisional position orientation PSE based on the history of past actual position orientation PSAs. The determination unit 31 outputs the provisional position orientation information of the hand HN obtained by the determination to the drawing unit 32 and the communication unit 50. The drawing unit 32 controls the display of the virtual object VOB corresponding to the hand HN based on the determined intermediate position orientation PSM.

[0125] In step S46, the determination unit 31 determines whether the first transition period has elapsed. If it is determined in step S46 that the first transition period has elapsed (step S46: Yes), the process proceeds to step S31, and the determination unit 31 performs the normal tracking state processing. After the first transition period has elapsed, and until the detection of the hand HN fails again, the determination unit 31 determines the actual positional orientation PSA of the hand HN detected based on the sensor information as the position and orientation of the hand HN.

[0126] If it is determined in step S46 that the first transition period has not elapsed (step S46: No), the process returns to step S44 and continues the transition process from steps S44 to S46 until the first transition period ends.

[0127] If it is determined in step S44 that the detection of hand HN has failed (step S44: No), the process proceeds to step S47. In step S47, the determination unit 31 determines whether or not the detection of object OB has been successful. If it is determined in step S47 that the detection of object OB has failed (step S47: No), the detection state of object OB transitions to the IDLE state. The determination unit 31 stops determining the position and orientation of object OB until it succeeds in detecting object OB.

[0128] If it is determined in step S47 that object OB has been successfully detected (step S47: Yes), the process proceeds to step S48. In step S48, the determination unit 31 determines whether the time during which the failure to detect hand HN continues has exceeded a predetermined time limit.

[0129] In step S48, if it is determined that the time during which the failure continues exceeds the time limit (step S48: Yes), the process proceeds to step S49. In step S49, the determination unit 31 notifies the drawing unit 32 and the communication unit 50 that the detection of the hand HN has failed. The detection state of the hand HN transitions to the IDLE state. The determination unit 31 stops determining the position and orientation of the hand HN until it succeeds in detecting the hand HN. The drawing unit 32 stops displaying the virtual object VOB corresponding to the hand HN until it succeeds in detecting the hand HN. In step S48, if it is determined that the time limit has not elapsed (step S48: No), the process returns to step S41.

[0130] Figure 15 shows an example of the processing flow during the waiting period for object OB to return to its original position when hand HN is gripping object OB.

[0131] In step S51, the determination unit 31 estimates the current position and orientation of object OB based on the history of the most recent object OB's actual position orientation PSA.

[0132] In step S52, the determination unit 31 outputs the provisional position and attitude information regarding the provisional position and attitude PSE of the object OB obtained by estimation to the drawing unit 32 and the communication unit 50.

[0133] In step S53, the determination unit 31 determines whether or not object OB has been detected. For example, if the determination unit 31 has acquired the actual position and orientation information of object OB from the information acquisition unit 20, it determines that detection of object OB has been successful. If the determination unit 31 has not acquired the actual position and orientation information of object OB from the information acquisition unit 20, it determines that detection of object OB has failed.

[0134] If it is determined in step S53 that object OB has been successfully detected (step S53: Yes), the process proceeds to step S54. In step S54, the determination unit 31 performs a process to transition to the normal tracking process.

[0135] If object OB is successfully detected, the detection state of object OB transitions to the tracking state. As shown in step S33, in the tracking state, the actual position and attitude PSA detected based on the sensor information is used as the position and attitude of object OB. However, if the actual position and attitude PSA is used as is, a large discrepancy may occur between it and the immediately preceding temporary position and attitude PSE due to position and attitude errors accumulated during the recovery waiting period. Therefore, the determination unit 31 sets a predetermined period immediately after the return to object OB detection as the first transition period, and gradually reduces the aforementioned discrepancy within the first transition period.

[0136] For example, during the first transition period, the determination unit 31 determines the current position and orientation of object OB by determining the intermediate position orientation PSM between the current actual position orientation PSA of object OB based on sensor information acquired by the body information acquisition unit 22 and the current provisional position orientation PSE of object OB based on the history of past actual position orientation PSAs of object OB. The determination unit 31 outputs the provisional position orientation information of object OB obtained by the determination to the drawing unit 32 and the communication unit 50. The drawing unit 32 controls the display of the virtual object VOB corresponding to object OB based on the determined intermediate position orientation PSM.

[0137] In step S55, the determination unit 31 determines whether the first transition period has elapsed. If it is determined in step S55 that the first transition period has elapsed (step S55: Yes), the process proceeds to step S31, and the determination unit 31 performs the normal tracking state processing. After the first transition period has elapsed, and until the detection of object OB fails again, the determination unit 31 determines the actual position and orientation PSA of object OB detected based on the sensor information as the position and orientation of object OB.

[0138] If it is determined in step S55 that the first transition period has not elapsed (step S55: No), the process returns to step S53 and continues the transition process from steps S53 to S55 until the first transition period ends.

[0139] If it is determined in step S53 that the detection of object OB has failed (step S53: No), the process proceeds to step S56. In step S56, the determination unit 31 determines whether or not the detection of hand HN has been successful. If it is determined in step S56 that the detection of hand HN has failed (step S56: No), the detection state of hand HN transitions to a recovery waiting state. The determination unit 31 determines the position and orientation of hand HN based on the history of the actual position and orientation PSA of hand HN immediately before the failure to detect hand HN began.

[0140] If it is determined in step S56 that the detection of hand HN has been successful (step S56: Yes), the process proceeds to step S57. In step S57, the determination unit 31 determines whether the time during which the failure to detect object OB continues has exceeded a predetermined time limit.

[0141] In step S57, if it is determined that the time during which the failure continues exceeds the time limit (step S57: Yes), the process proceeds to step S58. In step S58, the determination unit 31 notifies the drawing unit 32 and the communication unit 50 that object OB detection has failed. The object OB detection state transitions to the IDLE state. The determination unit 31 stops determining the position and orientation of object OB until object OB detection is successful. The drawing unit 32 stops displaying the virtual object VOB corresponding to object OB until object OB detection is successful. In step S57, if it is determined that the time limit has not elapsed (step S57: No), the process returns to step S51.

[0142] [2-2. Effects] In this embodiment, the determination of the position and orientation of object OB by the determination unit 31 is performed only when object OB is being held by the hand HN. An object OB held by the hand HN is expected to be an object of high attention. By increasing the continuity of the position and orientation of such an object OB, a display with less sense of incongruity can be obtained.

[0143] [3. Third Embodiment] [3-1. Information Processing Methods] Figure 17 illustrates an example of information processing in the third embodiment.

[0144] The difference in this embodiment from the first embodiment is that the display control unit 30 causes avatar AB to perform a pre-set autonomous operation during the recovery waiting period. The display control unit 30 sets a predetermined period immediately after the hand HN detection begins to fail as the second transition period. The display control unit 30 determines the position and orientation of the hand HN during the second transition period to be an intermediate position orientation PSM between the position and orientation of the hand HN corresponding to the autonomous operation and the provisional position orientation PSE of the hand HN estimated based on the history of the actual position orientation PSA of the hand HN immediately before the hand HN detection began to fail. Based on the determined position and orientation of the hand HN, the display control unit 30 controls the display of the virtual object VOB corresponding to the hand HN.

[0145] For example, time t1 indicates the time when the detection of the hand name began to fail. The determination unit 31 sets the period from time t1 to time t3 as the second transition period.

[0146] At time t1, the determination unit 31 calculates the provisional position and orientation PSE of the hand HN at time t1 based on the history of the hand HN's actual position and orientation PSA in the past. The determination unit 31 obtains the position and orientation PST at time t1 as defined by autonomous operation. The determination unit 31 determines that the intermediate position and orientation PSM1 between the position and orientation PST at time t1 as defined by autonomous operation and the provisional position and orientation PSE at time t1 calculated based on the history of the hand HN's actual position and orientation PSA is the position and orientation of the hand HN at time t1.

[0147] At time t2, the determination unit 31 obtains the position and orientation PST at time t2 as defined by autonomous operation. The determination unit 31 determines that the intermediate position and orientation PSM2 between the position and orientation PST at time t2 as defined by autonomous operation and the intermediate position and orientation PSM1 at time t1 obtained by the determination at time t1 is the position and orientation of the hand HN at time t2.

[0148] At time t3, the determination unit 31 obtains the position and orientation PST at time t3 as defined by autonomous operation. The determination unit 31 determines that the intermediate position and orientation PSM3 between the position and orientation PST at time t3 as defined by autonomous operation and the intermediate position and orientation PSM2 at time t2 obtained by the determination at time t2 is the position and orientation of the hand HN at time t3.

[0149] From time t4 onward, the determination unit 31 uses the position and orientation PST determined by autonomous operation as the position and orientation of hand HN.

[0150] [3-2. Effects] In this embodiment, the discontinuity in the position and orientation of the hand HN before and after the failure to detect it is made less noticeable by autonomous operation. After the failure to detect the hand HN, its position and orientation gradually change to match the position and orientation of the hand HN according to the autonomous operation. Therefore, the continuity of position and orientation is increased.

[0151] [Note] Furthermore, this technology can also be configured as follows. (1) A method of information processing performed by a computer, Receiving first information regarding the position and orientation of a first user from a first information processing device, Receiving second information relating to a first physical object from the first information processing device, In the first state, a first avatar corresponding to the first user is displayed based on the first information, In the first state, a virtual object corresponding to the first physical object is displayed based on the second information, Receiving first failure information from the first information processing device indicating a failure to detect the first physical object, In the second state, the first avatar is displayed based on third information relating to the position and orientation of the first user, In the second state, the display of the virtual object is stopped based on the first failure information, Includes, The first physical object is an object grasped by the first user. Information processing methods. (2) The information processing method described in (1) above, wherein the second information includes the position and orientation of the first physical object. (3) The information processing method described in (1) above, wherein the first failure information is output from the first information processing device based on the fact that the duration of the failure to detect the first physical object exceeds a predetermined time limit. (4) The information processing method described in (1) above, wherein the second state is a state in which the detection of the first physical object has failed and the detection of a part of the first user's body has succeeded. (5) The information processing method according to (1) above, wherein the first avatar includes a virtual object corresponding to the head of the first user and a virtual object corresponding to a part of the body of the first user. (6) A program that causes a computer to execute the information processing method described in (1) above. (7) A head information acquisition unit that acquires the position and orientation of the user's head, A body information acquisition unit that acquires the position and posture of a part of the user's body other than the head, A determination unit that acquires the position and orientation of the actual object grasped by the part of the body, A communication unit that transmits information regarding the position and orientation of the head, information regarding the position and orientation of a part of the body, and information regarding the position and orientation of the physical object to another information processing device, It has, The determination unit determines whether the part of the body is grasping the actual object, The aforementioned communications unit is If it is determined that the part of the body is grasping the physical object, and the detection of the physical object fails, failure information indicating the failure to detect the physical object is transmitted to the other information processing device. If it is determined that the part of the body is not grasping the physical object, and the detection of the physical object fails, the failure information will not be transmitted. Information processing device. (8) The information processing device according to (7) above, wherein the determination unit determines that a part of the body is grasping the physical object, and when it fails to detect the physical object, it transitions the detection state of the physical object to a recovery waiting state. (9) The information processing device according to (7) above, wherein the determination unit determines that the part of the body is not grasping the physical object and fails to detect the physical object, and stops determining the position and orientation of the physical object. (10) The information processing device according to (7) above, wherein the communication unit transmits the failure information to the other information processing device when the time during which the failure to detect the physical object continues exceeds a predetermined time limit. (11) The information processing device described in (7) above, wherein the part of the body is a hand. (12) The information processing device according to (7) above, wherein the determination unit determines that the part of the body is grasping the real object when the distance between the part of the body and the real object is smaller than a preset threshold. [Explanation of Symbols]

[0152] 21 Head information acquisition unit 22 Physical Information Acquisition Department 31 Judgment section 50 Communications Department 1000 computers AB Avatar HN Hand (part of the body) OB object (physical object) TM terminal (information processing device) TM1 First Information Processing Device TM2 Second Information Processing Device U User U1 First User U2 Second User VOB (Virtual Object)

Claims

1. A method of information processing performed by a computer, Receiving first information regarding the position and orientation of a first user from a first information processing device, Receiving second information relating to a first physical object from the first information processing device, In the first state, a first avatar corresponding to the first user is displayed based on the first information, In the first state, a virtual object corresponding to the first physical object is displayed based on the second information, The first information processing device receives first failure information indicating a failure to detect the first physical object, In the second state, the first avatar is displayed based on the third information relating to the position and orientation of the first user, In the second state, the display of the virtual object is stopped based on the first failure information, Includes, The first physical object is an object grasped by the first user. Information processing methods.

2. The information processing method according to claim 1, wherein the second information includes the position and orientation of the first physical object.

3. The information processing method according to claim 1, wherein the first failure information is output from the first information processing device based on the fact that the time during which the failure to detect the first physical object continues exceeds a predetermined time limit.

4. The information processing method according to claim 1, wherein the second state is a state in which the detection of the first physical object has failed and the detection of a part of the first user's body has succeeded.

5. The information processing method according to claim 1, wherein the first avatar includes a virtual object corresponding to the head of the first user and a virtual object corresponding to a part of the body of the first user.

6. A program that causes a computer to execute the information processing method described in claim 1.

7. A head information acquisition unit that acquires the position and orientation of the user's head, A body information acquisition unit that acquires the position and posture of a part of the user's body other than the head, A determination unit that acquires the position and orientation of the actual object grasped by the part of the body, A communication unit that transmits information regarding the position and orientation of the head, information regarding the position and orientation of a part of the body, and information regarding the position and orientation of the physical object to another information processing device, It has, The determination unit determines whether the part of the body is grasping the actual object, The aforementioned communications unit is If it is determined that the part of the body is grasping the physical object, and the detection of the physical object fails, failure information indicating the failure to detect the physical object is transmitted to the other information processing device. If it is determined that the part of the body is not grasping the physical object, and the detection of the physical object fails, the failure information will not be transmitted. Information processing device.

8. The information processing apparatus according to claim 7, wherein the determination unit determines that a part of the body is grasping the physical object, and when it fails to detect the physical object, it transitions the physical object detection state to a recovery waiting state.

9. The information processing device according to claim 7, wherein the determination unit stops determining the position and orientation of the actual object when it determines that the part of the body is not grasping the actual object and fails to detect the actual object.

10. The information processing apparatus according to claim 7, wherein the communication unit transmits the failure information to the other information processing apparatus when the time during which the failure to detect the physical object continues exceeds a predetermined time limit.

11. The information processing apparatus according to claim 7, wherein the part of the body is a hand.

12. The information processing device according to claim 7, wherein the determination unit determines that the part of the body is grasping the real object when the distance between the part of the body and the real object is smaller than a preset threshold.

Images