Motion tracking device and program
The motion tracking device and program effectively address the limitations of SLAM by tracking and estimating the positions and postures of moving objects in real-time, even in overlapping scenarios, using advanced data processing and learning models.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- NTT DOCOMO INC
- Filing Date
- 2025-01-10
- Publication Date
- 2026-07-23
AI Technical Summary
Existing technologies, such as SLAM, are inadequate for real-time tracking of arbitrary moving objects, particularly in scenarios where multiple objects are in contact or overlapping.
A motion tracking device and program that acquire positional and posture information between moving bodies, track their positions and postures, and estimate the positions and postures of other moving bodies based on these relationships, using learning models and tracking units to manage virtual and real-world interactions.
Enables real-time tracking of moving objects, even when they are in contact or overlapping, by accurately estimating their positions and postures through advanced data processing and learning models.
Smart Images
Figure 2026121069000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a moving object tracking device and a program.
Background Art
[0002] In Patent Document 1, when tracking the position of a moving object by analyzing a moving image composed of a distance image in which only the distance information from the moving object such as an object moving in space to the imaging position is imaged, even if a plurality of moving objects are in contact or overlapping, a technique for distinguishing and tracking each moving object is disclosed.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] As another technique for tracking the position of an object, SLAM (Simultaneous Localization and Mapping) for realizing self-position estimation of an object and creation of an environmental map can be mentioned. However, while SLAM has few restrictions on the object to be tracked in terms of position, since detection of stationary objects such as map creation is the main application, it is not suitable for tracking moving objects. In particular, real-time tracking of the position of an arbitrary moving object cannot be practically solved only by SLAM.
Means for Solving the Problems
[0005] A motion tracking device according to one aspect of the present disclosure includes: an acquisition unit that acquires first information indicating at least the positional relationship in space between a first motion body and a second motion body whose position can change in accordance with the change in position of the first motion body, and second information for tracking the position of the first motion body in space; a tracking unit that tracks the position of the first motion body in space based on the second information; and an estimation unit that estimates the position of the second motion body in space based on the tracking result of the tracking unit and the first information.
[0006] A motion tracking device according to another aspect of the present disclosure includes: an acquisition unit that acquires first information showing the relationship between the posture of a first motion and the posture of a second motion in space, and second information for tracking the posture of a first motion in space, for a first motion and a second motion whose posture can change in accordance with the change in posture of the first motion; a tracking unit that tracks the posture of the first motion based on the second information; and an estimation unit that estimates the posture of the second motion based on the tracking result of the tracking unit's analysis of the posture of the first motion and the first information.
[0007] Furthermore, a program according to one aspect of the present disclosure causes a computer to function as an acquisition unit that acquires first information indicating at least the positional relationship in space between a first moving body and a second moving body whose position can change in accordance with the change in the position of the first moving body, and second information for tracking the position of the first moving body in space; a tracking unit that tracks the position of the first moving body in space based on the second information; and an estimation unit that estimates the position of the second moving body in space based on the tracking result of the tracking unit and the first information.
[0008] Furthermore, a program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires first information showing at least the relationship between the posture of a first moving body and the posture of a second moving body in space, and second information for tracking the posture of the first moving body in space, for a first moving body and a second moving body whose posture can change in accordance with the change in posture of the first moving body; a tracking unit that tracks the posture of the first moving body based on the second information; and an estimation unit that estimates the posture of the second moving body based on the tracking result of the tracking unit and the first information. [Effects of the Invention]
[0009] According to this disclosure, the position or orientation of any moving object can be tracked in real time. [Brief explanation of the drawing]
[0010] [Figure 1] This is a block diagram showing an example configuration of a motion tracking system 1 including a virtual space server 20A according to the first embodiment of the motion tracking device of the present disclosure. [Figure 2] This figure shows an example of a door 40 corresponding to a virtual door for entering a virtual space. [Figure 3] This figure shows an example of a virtual door VD displayed overlaid on door 40. [Figure 4] This diagram shows an example configuration of virtual space server 20A. [Figure 5] This figure shows an example of a management table TBL stored in the storage device 220 of the virtual space server 20A. [Figure 6] This diagram illustrates the estimation of the position of the door panel 42 by the estimation unit 230d of the virtual space server 20A. [Figure 7] This flowchart shows the processing flow in the reproduction method executed by the processing unit 230 of the virtual space server 20A according to the program PRA. [Figure 8] This figure shows an example of a trolley 50, which is another example of a second moving body. [Figure 9]This is a diagram illustrating the first and second moving bodies in a second embodiment of the present disclosure. [Figure 10] This figure shows an example configuration of the motion tracking device 20B according to the second embodiment of this disclosure. [Figure 11] This diagram illustrates the estimation of the drone 70's position by the estimation unit 230e of the motion tracking device 20B. [Figure 12] This is a flowchart showing the processing flow in the reproduction method executed by the processing unit 230 of the motion tracking device 20B according to the program PRB. [Modes for carrying out the invention]
[0011] (A. First Embodiment) Figure 1 shows an example configuration of a motion tracking system 1 including a virtual space server 20A according to one embodiment of the motion tracking device of the present disclosure. The virtual space server 20A is a device that provides virtual space services to user U. As shown in Figure 1, in addition to the virtual space server 20A, the motion tracking system 1 includes a terminal 10 that communicates with the virtual space server 20A via a communication network NW, and a glasses device 30 connected to the terminal 10.
[0012] Terminal 10 acquires virtual space data related to the virtual space by communicating with the virtual space server 20A via the communication network NW. Specific examples of virtual space data include image data representing a virtual door for entering the virtual space, and image data representing the scenery of the virtual space to which entry is made.
[0013] Terminal 10 is, for example, a smartphone or a tablet device. In this embodiment, terminal 10 is connected to the glasses device 30 by a wired connection, but it may also be connected to the glasses device 30 by a wireless connection. Terminal 10 displays images related to the virtual space on the glasses device 30 based on virtual space data acquired from the virtual space server 20A. Images related to the virtual space include an image representing a virtual door for entering the virtual space, and an image of the scenery of the virtual space to which entry is made.
[0014] The glass device 30 is a glasses-type display device. The glass device 30 is a transmissive HMD that transmits light and displays an image. The glass device 30 includes temples 91 and 92, a rim 93, body parts 94 and 95, and lenses 2aL and 2aR.
[0015] In the body part 94, a display panel for the left eye and an optical member for the left eye are provided. The display panel is, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display panel for the left eye displays an image related to the virtual space, for example, based on control from the terminal 10. The optical member for the left eye is an optical member that guides the light emitted from the display panel for the left eye to the lens 2aL.
[0016] In the body part 95, a display panel for the right eye and an optical member for the right eye are provided. The display panel for the right eye displays an image related to the virtual space, for example, based on control from the terminal 10. The optical member for the right eye is an optical member that guides the light emitted from the display panel for the right eye to the lens 2aR. <了
[0017] Each of the lenses 2aL and 2aR has a half mirror. The half mirror of the lens 2aL guides the light representing the real space to the left eye B1L of the user U by transmitting the light representing the real space. Also, the half mirror of the lens 2aL reflects the light guided by the optical member for the left eye to the left eye B1L of the user U. The half mirror of the lens 2aR guides the light representing the real space to the right eye BIR of the user U by transmitting the light representing the real space. Also, the half mirror of the lens 2aR reflects the light guided by the optical member for the right eye to the right eye B1R of the user U.
[0018] Light representing the real world passes through lenses 2aL and 2aR. When user U wears the glasses device 30 on their head, lens 2aL is positioned in front of user U's left eye B1L, and lens 2aR is positioned in front of user U's right eye B1R. User U, wearing the glasses device 30, can see both the real space represented by the light that has passed through lenses 2aL and 2aR, and the image of the virtual space displayed on lenses 2aL and 2aR. For example, user U can see an image of the virtual space superimposed on the real space.
[0019] A camera 96L is provided near lens 2aL on the rim 93, and a camera 96R is provided near lens 2aR on the rim 93. Both cameras 96L and 96R are RGB cameras. Cameras 96L and 96R form a stereo camera for acquiring an image of the field of view of user U wearing the glasses device 30 on their head (hereinafter referred to as the field of view image) and depth information regarding objects present in the field of view (for example, the distance from the center of the rim 93 on the glasses device 30). Cameras 96L and 96R are examples of measuring devices in this disclosure, but a LiDAR sensor may also be used as the measuring device. When a LiDAR sensor is used as the measuring device, if the brightness of the real space is below a predetermined threshold, depth information may be acquired using the LiDAR sensor, and if the brightness of the real space is above the predetermined threshold, depth information may be acquired using the stereo camera. Cameras 96L and 96R are also used for detecting and positioning AR markers that serve as landmarks when displaying a virtual door for entering the virtual space.
[0020] In this embodiment, the real space is the living room of user U, who uses terminal 10 and glasses device 30. In this real space, a door 40 corresponding to a virtual door for entering the virtual space is installed. The AR marker described above is attached to door 40. Door 40 is constructed to resemble a real door. Figure 2 shows an example of the configuration of door 40. Door 40 has a frame made up of a combination of prismatic members 41L, 41R, 41B, and 41T, and a door panel 42 that is rotatably connected to member 41R by hinges 43a and 43b. A doorknob 44 is provided on the door panel 42. User U can open door 40 by grasping the doorknob 44 and pulling it towards them. In addition, an AR marker 45 is provided in the center of door panel 42, which serves as a marker when displaying the virtual door.
[0021] As will be explained in more detail later, the virtual space server 20A stores virtual door information, which represents an image of the virtual door to be displayed overlaid on the door 40 to which the AR marker 45 is attached, and landscape information, which represents an image of the virtual space scenery to be entered by passing through this virtual door, associated with identification information indicating the AR marker 45. A specific example of the identification information indicating the AR marker 45 is the image of the AR marker 45. When user U puts the glasses device 30 on their head and performs an operation to instruct terminal 10 to start using the virtual space service, terminal 10 instructs cameras 96L and 96R to capture a field of view image and transmits image data representing the field of view image captured by cameras 96L and 96R to the virtual space server 20A. The virtual space server 20A analyzes the field of view image received from terminal 10, and when it detects that the field of view image contains an image of the AR marker 45, it transmits virtual door information corresponding to the AR marker 45 to terminal 10 and instructs terminal 10 to display the image represented by the virtual door information at the position corresponding to the AR marker 45. In response to this instruction, terminal 10 displays an image of the virtual door on the glasses device 30, so that the virtual door VD is displayed superimposed on the door 40 to the user U's eyes, as shown in Figure 3.
[0022] As described above, when user U grasps the doorknob 44 and pulls it towards him, the door panel 42, which is rotatably connected to member 41R by hinges 43a and 43b, rotates and the door 40 opens. Here, the position of user U's hand can be tracked using existing technology based on the output data from cameras 96L and 96R. The output data from cameras 96L and 96R represents a field of view image and is an example of measurement data representing the distance from an object within user U's field of view. In the motion tracking system 1, based on the position tracking result of a first motion (in this embodiment, user U's hand) whose position can be tracked using existing technology, the system tracks the change in position of a second motion (in this embodiment, the door panel 42) which is connected to a third motion (in this embodiment, the doorknob 44) whose position can change in accordance with the change in position of the first motion, and controls the display of the virtual door VD according to the change in position of the second motion.
[0023] Figure 4 is a block diagram showing an example configuration of the virtual space server 20A. As shown in Figure 4, the virtual space server 20A includes a communication device 210, a storage device 220, a processing device 230, and a bus 240 that connects these devices to each other.
[0024] The communication device 210 includes a communication interface circuit. The communication device 210 is connected to a communication network NW by wire or wireless connection. The communication device 210 communicates with other devices via the communication network NW. A specific example of another device that communicates with the communication device 210 is the terminal 10.
[0025] The storage device 220 is a recording medium that can be read by the processing device 230. The storage device 220 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory). The storage device 220 pre-stores a learning model MDL, a management table TBL, and a program PRA.
[0026] The MDL learning model is a machine learning model that has learned the amount of change in the position of the doorknob 44 relative to the frame of the door 40, corresponding to the degree of opening and closing of the door 40. The degree of opening and closing of the door 40 is represented by the angle θ that the door panel 42 makes with respect to the frame of the door 40. Specifically, θ = 0° means that the door 40 is closed, and the closer the value of angle θ is to 90°, the wider the door 40 is open. In this embodiment, the position of the frame of the door 40 is fixed, and the relative position of the doorknob 44 with respect to the door panel 42 is also fixed. The learning results of the MDL learning model reflect the relative position of the doorknob 44 with respect to the door panel 42, and the angle θ of rotation of the door panel 42 with respect to the frame is uniquely determined according to the amount of change in the position of the doorknob 44 with respect to the frame. The MDL learning model has learned the amount of change in the position of the doorknob 44 with respect to the frame, corresponding to multiple angles in 1° increments in the range of 0 to 90°. When the learning model MDL receives input on the change in the position of the doorknob 44 relative to the frame of the door 40, it outputs an angle θ (the angle θ that the door panel 42 makes with the frame of the door 40) that represents the degree of opening and closing of the door 40.
[0027] Figure 5 shows an example of the data structure of the management table TBL. As shown in Figure 5, the management table TBL stores virtual door information corresponding to each angle θ in 1° increments within the range of 0 to 90°, and landscape information, associated with identification information indicating the AR marker 45 attached to the door 40 corresponding to the virtual door. The virtual door information corresponding to angle θ is image data representing the image of the virtual door when it is open by angle θ.
[0028] The processing unit 230 includes one or more CPUs (Central Processing Units). One or more CPUs are an example of one or more processors. Each of the processors and CPUs is an example of a computer. For example, when the virtual space server 20A is powered on, the processing unit 230 reads the program PRA from the storage device 220. Then, the processing unit 230 executes the program PRA read from the storage device 220. The processing unit 230, operating according to the program PRA, communicates with the terminal 10 using the communication device 210. When the processing unit 230 receives field of view image data using the communication device 210, it determines whether or not the image of the AR marker 45 is included in the image represented by this field of view image data. If the image of the AR marker 45 is included in the image represented by the field of view image data, the processing unit 230 reads the virtual door information (θ=0°) stored in the management table TBL in association with the identification information of the AR marker 45, and instructs the terminal 10 to display the image represented by the virtual door information, i.e., the image of the virtual door in a closed state, at the position of the AR marker 45.
[0029] Furthermore, the processing unit 230 functions as an acquisition unit 230a, a detection unit 230b, a tracking unit 230c, and an estimation unit 230d, as shown in Figure 4, triggered by the detection of an image of the AR marker 45 from the image represented by the visual field image data. The functions of each of the acquisition unit 230a, detection unit 230b, tracking unit 230c, and estimation unit 230d are as follows.
[0030] The acquisition unit 230a communicates with the terminal 10 using the communication device 210 to represent an image of the space including the first moving object and the second moving object, and acquires measurement data (i.e., output data of cameras 96R and 96L) from the terminal 10 indicating the first distance from the first moving object to the measuring device (in this embodiment, cameras 96L and 96R) and the second distance from the second moving object to the measuring device. As described above, in this embodiment the first moving object is the hand of user U, and the second moving object is the door panel 42. The image of the space including the first moving object and the second moving object represents the positional relationship between the first moving object and the second moving object in space. The measurement data including this image is, in this disclosure, an example of first information that shows at least the positional relationship in space between the first moving object and the second moving object, whose position may change in accordance with the change in the position of the first moving object. As will be described in detail later, in this embodiment the position tracking of the first moving object is performed based on the measurement data. Measurement data is also an example of secondary information used to track the position of a first moving object in space.
[0031] The detection unit 230b detects a first state in which the second moving object comes into contact with the first moving object, based on the measurement data acquired by the acquisition unit 230a. In this embodiment, the detection unit 230b detects a first state in which the door panel 42 comes into contact with the first moving object, when the image represented by the measurement data acquired by the acquisition unit 230a shows that the hand of user U and the door knob 44 provided on the door panel 42 overlap, and the difference between the distance to user U's hand represented by the measurement data and the distance to the door knob 44 represented by the measurement data is less than a predetermined threshold (for example, 1 cm). The detection unit 230b then detects a first state in which the door panel 42 comes into contact with user U's hand via the door knob 44, i.e., the second moving object comes into contact with the first moving object. The detection unit 230b then uses the position of the first moving object in real space at the time the first state is detected as the reference position and writes reference position information indicating the reference position to the storage device 220.
[0032] The tracking unit 230c tracks the position of the first moving object in space based on the measurement data acquired by the acquisition unit 230a. As mentioned above, the first moving object in this embodiment is the hand of user U, and existing technologies such as Leap Motion may be used as appropriate for tracking the position of user U's hand based on the measurement data.
[0033] While the first state is maintained, the estimation unit 230d estimates the spatial position of the second moving body based on the tracking result of the tracking unit 230c for the position of the first moving body. In this embodiment, the estimation unit 230d first estimates the position of the doorknob 44 in space based on the tracking result of the position of the user U's hand. Specifically, the estimation unit 230d estimates the position of the doorknob 44 to be the same as the position of the user U's hand. Next, the estimation unit 230d calculates the amount of change in the position of the doorknob 44 based on the difference between the position of the doorknob 44 estimated based on the position of the user U's hand and the reference position indicated by the aforementioned reference position information. Then, the estimation unit 230d inputs this amount of change into the learning model MDL to estimate the position of the door panel 42 (the rotation angle θ of the door panel 42).
[0034] For example, as shown in Figure 6, at time T1, contact between user U's hand UH and the doorknob 44 is detected, and during the period from time T1 to time T2, user U grasps the doorknob 44 with hand UH and pulls it towards them, so that the angle θ that the door panel 42 makes with the frame becomes 30°. During the period from time T1 to time T2, the first state in which user U's hand UH is in contact with the doorknob 44 is maintained, so during this period, the position of the door panel 42 (the angle θ that the door panel 42 makes with the frame) is estimated based on the tracking result of the position of user U's hand UH. The processing unit 230 reads virtual door information corresponding to the estimated angle θ from the management table TBL and instructs terminal 10 to display the image represented by the read virtual door information. As a result, in user U's field of view, the virtual door VD opens in accordance with the action of grasping the doorknob 44 with hand UH and pulling it towards them.
[0035] Furthermore, the processing unit 230, which operates according to the program PRA, performs a motion tracking method that clearly demonstrates the features of this embodiment. Figure 7 is a flowchart showing the processing flow in this motion tracking method. As shown in Figure 7, this motion tracking method includes the following processes: acquisition process SA110, detection process SA120, tracking process SA130, and estimation process SA140.
[0036] In the acquisition process SA110, the processing unit 230 functions as an acquisition unit 230a. In the acquisition process SA110, the processing unit 230 communicates with the terminal 10 using the communication device 210 to represent an image of the space including the first moving object and the second moving object, and acquires measurement data indicating the first distance from the first moving object to the measuring device and the second distance from the second moving object to the measuring device.
[0037] In detection process SA120, the processing unit 230 functions as a detection unit 230b. In detection process SA120, the processing unit 230 detects a first state in which a second moving object comes into contact with a first moving object, based on the measurement data acquired in acquisition process SA110. In detection process SA120 in this embodiment, contact between the hand UH of user U and the doorknob 44 provided on the door panel 42 is detected.
[0038] In tracking process SA130, the processing unit 230 functions as a tracking unit 230c. In tracking process SA130, the processing unit 230 tracks the position of the first moving object (user U's hand UH) in space based on the measurement data acquired in acquisition process SA110.
[0039] In estimation process SA140, the processing unit 230 functions as an estimation unit 230d. In estimation process SA140, the processing unit 230 estimates the position of the third moving object (door handle 44) based on the tracking result of the first moving object (user U's hand UH) performed in tracking process SA130, and uses the learning model MDL to estimate the position of the second moving object in the first state based on the position estimation result for the third moving object and the relative position of the third moving object with respect to the second moving object (door panel 42).
[0040] According to this embodiment, it becomes possible to track the position of any second moving body whose position can change in accordance with the change in the position of the first moving body, based on the position tracking results of the first moving body whose position can be tracked based on measurement data. In this embodiment, the first moving body is the user U's hand UH, the second moving body is the door panel 42, and the third moving body is the doorknob 44. However, the second moving body in this disclosure is not limited to the door panel 42, and the third moving body is not limited to the doorknob 44. For example, the second moving body may be the trolley 50 shown in Figure 8, and the third moving body may be the handle 51 provided on the trolley 50. This is because, in the process of the user U transporting luggage using the trolley 50, the user U's hand UH is generally in contact with the handle 51, and the fact that the relative position of the handle 51 with respect to the trolley 50 is fixed is the same as the fact that the relative position of the doorknob 44 with respect to the door panel 42 is fixed.
[0041] In the above embodiment, a third moving body that can contact the first moving body was connected to a second moving body whose position can change in accordance with the change in the position of the first moving body. However, a fourth moving body that can contact the second moving body, which can change in position in accordance with the change in the position of the first moving body, may be connected to the first moving body. A specific example of this embodiment is a configuration in which the first moving body is the torso of another user different from user U, the second moving body is a chair with armrests that the other user can sit on, and the fourth moving body is the elbow of the other user. In this embodiment, the detection unit 230b should detect a first state in which the armrest of the chair contacts the elbow of the other user, that is, the chair contacts the torso of the other user, when the elbow of the other user and the armrest of the chair overlap in the image represented by the measurement data acquired by the acquisition unit 230a, and the difference between the distance to the elbow represented by the measurement data and the distance to the armrest represented by the measurement data falls below a predetermined threshold. The estimation unit 230d then estimates the position of the elbow in space based on the tracking results of the position of the other user's torso and the relative position of the elbow to the torso, and then estimates the position of the chair in the first state based on the estimation results of the position of the elbow.
[0042] (B: Second embodiment) In the first embodiment, in a first state in which a second moving object is in contact with a first moving object whose position can be tracked based on measurement data, the position of the second moving object was tracked based on the position tracking results for the first moving object. However, contact between the first and second moving objects is not necessarily required, and if the second moving object is an object whose position can change in accordance with the change in the position of the first moving object, it may be possible to track the position of the second moving object according to this disclosure. Figure 9 is a diagram illustrating the first and second moving objects in the second embodiment of this disclosure. In this embodiment, the first moving object is the user V, and the second moving object in this disclosure is a drone 70 flying in real space. While flying in real space, the drone 70 captures an image of an imaging area AR centered on position PA, the intersection point of a straight line drawn from the position of the drone 70 along the vertical axis to the ground surface and the ground surface. In this embodiment, the position of the drone 70 is fixed at a height H vertically above position PA. In other words, the relative position of the imaging area AR with respect to the drone 70 is fixed. Furthermore, in this embodiment, position PA is determined in accordance with changes in the position of user V, such that its relative position as seen from position PV of user V remains constant. In this embodiment, user V is an example of a first moving body, drone 70 is an example of a second moving body, and imaging area AR is an example of a fifth moving body whose position changes in accordance with changes in the position of the first moving body, while its relative position to the second moving body is fixed.
[0043] Figure 10 shows an example configuration of the motion tracking device 20B according to a second embodiment of the present disclosure. The motion tracking device 20B, together with the aforementioned glasses device 30 and terminal 10, constitutes a motion tracking system for tracking the position of a drone 70. In Figure 10, the same components as those in Figure 4 are denoted by the same reference numerals. As is clear from comparing Figure 10 and Figure 4, the hardware configuration of the motion tracking device 20B is the same as the hardware configuration of the virtual space server 20A. The configuration of the motion tracking device 20B differs from the configuration of the virtual space server 20A in two respects: the management table TBL and the learning model MDL are not stored in the storage device 220, and the program PRB is stored in the storage device 220 instead of the program PRA.
[0044] Program PRB differs from Program PRA in that it causes the processing unit 230 to function as an acquisition unit 230a, a tracking unit 230c, and an estimation unit 230e. The acquisition unit 230a, as in the first embodiment, communicates with the terminal 10 using the communication device 210 to represent an image of the space including the first moving object and the second moving object, and acquires measurement data from the terminal 10 indicating the first distance from the first moving object to the measuring device and the second distance from the second moving object to the measuring device. The tracking unit 230c tracks the position of the first moving object (user V in this embodiment) in space based on the measurement data acquired by the acquisition unit 230a. Existing technologies may be used as appropriate for tracking the position of user V based on the measurement data.
[0045] The estimation unit 230e tracks the position of the imaging area AR (specifically, position PA) based on the position tracking results for user V. Then, based on the position tracking results for the imaging area AR and the relative position of the imaging area AR with respect to the drone 70, the estimation unit 230e estimates the position of the drone 70 to be a position vertically upward from point PA at a height H.
[0046] For example, as shown in Figure 11, suppose that at time T1, user V is located at position PV1(x1,y1), and that user V moved to position PV2(x1+Δx,y1) during the period from time T1 to time T2. Note that the x-axis in Figure 11 corresponds to latitude, the y-axis corresponds to longitude, etc., and these are coordinate axes that define the position on the Earth's surface in real space. In this case, if the center of the imaging area AR at time T1 is at position PA1(x2,y2), then the relative position of the center of the imaging area AR with respect to user V's position remains unchanged, so the center of the imaging area AR at time T1 is at position PA2(x2+Δx,y2). Therefore, the estimation unit 230e estimates the position of the drone 70 at time T2 to be at a height H vertically above position PA2(x2+ΔX,y2). Note that if vertical position information such as altitude can be obtained for positions PA1 and PA2, the position of the drone 70 may be estimated by taking this position information into account.
[0047] Furthermore, the processing unit 230, which operates according to the program PRB, executes a motion tracking method that clearly demonstrates the features of this embodiment. Figure 12 is a flowchart showing the processing flow in this motion tracking method. As shown in Figure 12, this motion tracking method includes the acquisition process SA110, the tracking process SA130, and the estimation process SB140.
[0048] In acquisition process SA110, the processing unit 230 functions as an acquisition unit 230a. In acquisition process SA110, the processing unit 230 communicates with the terminal 10 using the communication device 210 to represent an image of the space including the first moving object (user V) and the second moving object (drone 70), and acquires measurement data indicating the first distance from the first moving object to the measurement device and the second distance from the second moving object to the measurement device. In tracking process SA130, the processing unit 230 functions as a tracking unit 230c. In tracking process SA130, the processing unit 230 tracks the position of the first moving object in space based on the measurement data acquired in acquisition process SA110.
[0049] In estimation process SB140, the processing unit 230 functions as an estimation unit 230e. In estimation process SB140, the processing unit 230 estimates the position of the fifth moving object (imaging area AR) based on the tracking result of the first moving object's position by tracking process SA130, and estimates the position of the second moving object based on the position estimation result for the fifth moving object and the relative position of the fifth moving object with respect to the second moving object.
[0050] This embodiment also makes it possible to track the position of any second moving object whose position can change in accordance with the change in the position of the first moving object, based on the position tracking results for the first moving object, whose position can be tracked based on measurement data. Furthermore, according to this embodiment, even if the second moving object is not in contact with the first moving object, it is possible to track the position of any second moving object whose position can change in accordance with the change in the position of the first moving object, based on the position tracking results for the first moving object.
[0051] (C: Transformation) Each of the above embodiments can be modified as follows. (C-1: Torture 1) In each of the above embodiments, the terminal 10 was a smartphone or a tablet device, but it may also be a stationary personal computer. Furthermore, in each of the above embodiments, the glasses device 30 was a transmissive HMD that transmits light from the real world and displays an image, but it may also be an opaque HMD that does not transmit light from the real world and displays an image of the real world captured by an imaging device to the user. In addition, in the second embodiment, the glasses device 30 is not essential, and instead of the glasses device 30, a measuring device that outputs measurement data representing an image of space including a first moving object and a second moving object whose position can change in accordance with the change in the position of the first moving object, as well as a first distance between the first moving object and the second moving object and the second distance to the second moving object may be connected to the terminal 10.
[0052] (C-2: Variation 2) In the first embodiment described above, a door 40 that mimicked a real door was used as the real door corresponding to the virtual door. However, it could also be a real door installed in the real space where user U is located (for example, if the real space is user U's room, it could be a door for entering and exiting that room).
[0053] (C-3: Modification 3) The images included in the measurement data in each of the above embodiments represent the spatial positional relationship between a first moving body and a second moving body whose position can change in accordance with the change in the position of the first moving body. However, these images also represent three-dimensional relationships other than positional relationships, such as the relationship between the posture of the first moving body and the posture of the second moving body. By using the relationship between the posture of the first moving body and the posture of the second moving body, and using information for tracking the posture of the first moving body in space as second information, it becomes possible to estimate the posture of the second moving body based on the posture of the first moving body. The above images are a concrete example of this second information.
[0054] For example, if the first moving object is the user U's hand UH, the second moving object is a trolley 50, and the third moving object is a handle 51 provided on the trolley 50, it becomes possible to estimate the orientation of the trolley 50 (i.e., the orientation of the trolley 50) based on the angle of the user U's hand UH relative to the handle 51 (i.e., the orientation of the hand UH). This is because when the trolley 50 is moved by pushing the handle 51 with the hand UH, the orientation of the trolley 50 changes depending on the angle of the hand UH. Also, if the first moving object is the shape of the user's face and the second moving object is the user's head, if the position of a camera, for example, which is used to capture the first moving object, is known, the position of the second moving object can be estimated from that camera. This is because the size of the eye is anatomically known, so if the size of the eye in the face captured in the image taken by the camera is known, the distance from the camera to the eye can be calculated.
[0055] Furthermore, by using both the positional relationship between the first and second moving bodies, as well as the relationship between the posture of the first and second moving bodies, it becomes possible to estimate the position of the second moving body based on the position of the first moving body, and also to estimate the posture of the second moving body based on the posture of the first moving body. In addition, by using both hands of user U as the first moving body, the estimation accuracy regarding the position of the second moving body is improved, as is the estimation accuracy regarding the posture of the second moving body. In other words, the more methods there are for obtaining the three-dimensional relationship between the first and second moving bodies, the higher the estimation accuracy regarding the second moving body becomes. Note that the first information in this disclosure can be any information that represents some three-dimensional relationship between the first and second moving bodies, and is not limited to an image of the space in which the first and second moving bodies exist.
[0056] (C-4: Modification 4) In the first embodiment described above, the program PRA was pre-stored in the storage device 220 of the virtual space server 20A, but the program PRA may be manufactured or sold separately. When selling the program PRA, possible methods of providing it to the buyer include writing the program PRA to a computer-readable recording medium such as flash ROM and distributing it, or distributing it by download via a telecommunications line. The program PRB may also be manufactured or sold separately in a similar manner.
[0057] (C-5: Variation 5) In the first embodiment described above, the acquisition unit 230a, detection unit 230b, tracking unit 230c, and estimation unit 230d may be composed of electronic circuits such as ASICs. A virtual space server 20A may be configured by combining these electronic circuits. Similarly, in the second embodiment described above, the acquisition unit 230a, tracking unit 230c, and estimation unit 230e may be composed of electronic circuits such as ASICs, and a motion tracking device 20B may be configured by combining these electronic circuits.
[0058] (D: Other) (1) In the embodiments described above, ROM and RAM were given as examples of the storage device 220, but the storage device 220 may be a flexible disk, magneto-optical disk (e.g., compact disk, digital multipurpose disk, Blu-ray® disk), smart card, flash memory device (e.g., card, stick, key drive), CD-ROM (Compact Disc-ROM), register, removable disk, hard disk, floppy® disk, magnetic strip, database, server, or other suitable storage medium.
[0059] (2) In each of the embodiments described above, the information, signals, etc. may be represented using any of the various different techniques. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.
[0060] (3) In each of the embodiments described above, the input and output information may be stored in a specific location (e.g., memory) or managed using a table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.
[0061] (4) In each of the embodiments described above, the determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).
[0062] (5) The processing procedures, sequences, flowcharts, etc., exemplified in each of the embodiments described above may be in any order, as long as they do not contradict each other. For example, in the methods described herein, the elements of various steps are presented using an exemplary order and are not limited to the specific order presented.
[0063] (6) Each function illustrated in Figure 4 or Figure 10 is implemented by any combination of at least one of hardware and software. Furthermore, the method of implementing each function block is not particularly limited. That is, each function block may be implemented using one device that is physically or logically coupled, or it may be implemented using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired, wireless, etc.). A function block may be implemented by combining the one or more devices with software.
[0064] (7) The programs illustrated in each of the embodiments described above should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages or by any other name.
[0065] Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.), at least one of these wired and wireless technologies is included in the definition of a transmission medium.
[0066] (8) In each of the above-mentioned forms, the terms “system” and “network” shall be used interchangeably.
[0067] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a given value, or other corresponding information.
[0068] (10) In each of the embodiments described above, the terms “connected,” “coupled,” or any variation thereof means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” with each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in this disclosure, two elements may be considered to be “connected” or “coupled” with each other using at least one of one or more wires, cables and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain and optical (both visible and invisible) domain, etc.
[0069] (11) In each of the embodiments described above, the phrase “based on” does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”
[0070] (12) The terms “determining” and “determining” as used in this disclosure may encompass a wide variety of actions. “Determining” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or other data structure), ascertaining, etc. “Determining” may also include, for example, receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), etc. Furthermore, "judgment" and "decision" can include considering something as having been "judged" or "decided" after resolving, selecting, choosing, establishing, comparing, etc. In other words, "judgment" and "decision" can include considering something as having been "judged" or "decided" after some action. Also, "judgment (decision)" can be reinterpreted as "assuming," "expecting," "considering," etc.
[0071] (13) Where the terms “include,” “including,” and variations thereof are used in the embodiments described above, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to be exclusive OR.
[0072] (14) Where articles are added in translation, for example, a, an, and the in English, the disclosure may include the fact that the noun following these articles is plural.
[0073] (15) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combined” may be interpreted in the same way as “different.”
[0074] (16) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed in practice. Furthermore, notification of certain information (e.g., notification that "X is") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).
[0075] (E: Aspects as understood from each of the above embodiments or modifications) Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Accordingly, the descriptions in the present disclosure are for illustrative purposes only and are not intended to be restrictive in any way. The following embodiments can be understood from at least one of the embodiments or modifications described above.
[0076] A motion tracking device according to a first aspect of the present disclosure includes: an acquisition unit that acquires first information indicating at least the positional relationship in space between a first motion and a second motion whose position can change in accordance with the change in position of the first motion, and second information for tracking the position of the first motion in space; a tracking unit that tracks the position of the first motion in space based on the second information; and an estimation unit that estimates the position of the second motion in space based on the tracking result of the tracking unit's position of the first motion and the first information. According to this aspect, it becomes possible to estimate the position of the second motion based on the position tracking result of the first motion, and thus it becomes possible to track the position of the second motion.
[0077] A motion tracking device according to a second aspect of the present disclosure (an example of the first aspect) includes a detection unit that detects a first state in which the first motion object comes into contact with the second motion object based on the measurement data, and the estimation unit in the motion tracking device of the second aspect estimates the position of the second motion object in space in the first state based on the tracking result of the tracking unit's position of the first motion object. According to this aspect, it becomes possible to estimate the position of the second motion object based on the tracking result of the position of the first motion object in the first state in which the first motion object comes into contact with the second motion object, and thus it becomes possible to track the position of the second motion object in the first state.
[0078] In a third aspect of this disclosure (an example of the second aspect), the two moving bodies are connected to a third moving body, the third moving body having a fixed relative position to the second moving body and making contact with the first moving body in the first state. The estimation unit in the moving body tracking device of the third aspect tracks the position of the third moving body in the first state based on the position tracking result for the first moving body, and estimates the position of the second moving body in the first state based on the position tracking result for the third moving body and the relative position of the third moving body with respect to the second moving body. According to this aspect, the position of the third moving body making contact with the first moving body can be tracked based on the position tracking result for the first moving body in the first state, and since the relative position of the third moving body with respect to the second moving body is fixed, the position of the second moving body can be estimated. Therefore, according to this aspect as well, the position of the second moving body in the first state can be tracked.
[0079] In the fourth aspect of this disclosure (an example of the second aspect), the first moving body is connected to a fourth moving body whose relative position to the first moving body is known and which the second moving body contacts in the first state. The estimation unit in the moving body tracking device of the fourth aspect tracks the position of the fourth moving body in the first state based on the position tracking result for the first moving body and the relative position of the fourth moving body to the first moving body, and estimates the position of the second moving body in the first state based on the position tracking result for the fourth moving body. According to this aspect, based on the position tracking result for the first moving body, it is possible to track the position of the fourth moving body connected to the first moving body and whose relative position to the first moving body is known, and based on the position tracking result for the fourth moving body, it is possible to estimate the position of the second moving body that contacts the fourth moving body in the first state. Therefore, according to this aspect as well, it is possible to track the position of the second moving body in the first state.
[0080] In the fifth aspect of this disclosure (an example of the first aspect), the space includes a fifth moving body whose position changes in accordance with the change in the position of the first moving body, while its relative position to the second moving body is fixed. The estimation unit in the motion tracking device of the fifth aspect tracks the position of the fifth moving body based on the position tracking result for the first moving body, and estimates the position of the second moving body based on the position tracking result for the fifth moving body and the relative position of the fifth moving body to the second moving body. According to this aspect, it becomes possible to estimate the position of the second moving body, whose relative position to the fifth moving body is fixed, based on the position tracking result of the fifth moving body, whose position changes in accordance with the change in the position of the first moving body. Therefore, it becomes possible to track the position of the second moving body according to this aspect as well.
[0081] A motion tracking device according to a sixth aspect of the present disclosure includes: an acquisition unit that acquires first information showing the relationship between the posture of a first motion and the posture of a second motion in space, and second information for tracking the posture of the first motion in space, for a first motion and a second motion whose posture can change in accordance with the change in posture of the first motion; a tracking unit that tracks the posture of the first motion based on the second information; and an estimation unit that estimates the posture of the second motion based on the tracking result of the tracking unit's analysis of the posture of the first motion and the first information. According to this aspect, it becomes possible to estimate the posture of the second motion based on the posture tracking result of the first motion, thus enabling the tracking of the posture of the second motion.
[0082] A program according to a seventh aspect of this disclosure causes a computer to function as: an acquisition unit that acquires first information indicating at least the positional relationship in space between a first moving body and a second moving body whose position can change in accordance with the change in the position of the first moving body, and second information for tracking the position of the first moving body in space; a tracking unit that tracks the position of the first moving body in space based on the second information; and an estimation unit that estimates the position of the second moving body in space based on the tracking result of the tracking unit's position of the first moving body and the first information. In this aspect as well, similar to the first aspect, it is possible to estimate the position of the second moving body based on the position tracking result of the first moving body, and thus it is possible to track the position of the second moving body.
[0083] A program according to the eighth aspect of this disclosure causes a computer to function as: an acquisition unit that acquires first information indicating the relationship between the posture of a first moving body and the posture of a second moving body in space, and second information for tracking the posture of the first moving body in space; a tracking unit that tracks the posture of the first moving body based on the second information; and an estimation unit that estimates the posture of the second moving body based on the tracking result of the tracking unit and the first information. In this aspect as well, similar to the sixth aspect, it becomes possible to estimate the posture of the second moving body based on the posture tracking result of the first moving body, thus enabling posture tracking of the second moving body. [Explanation of symbols]
[0084] 1...Motion tracking system, 10...Terminal, 20A...Virtual space server, 20B...Motion tracking device, 30...Glass device, 40...Door, 50...Cart, 70...Drone, NW...Communication network, 210...Communication device, 220...Storage device, 230...Processing device, 240...Bus, 230a...Acquisition unit, 230b...Detection unit, 230c...Tracking unit, 230d,230e...Estimation unit, PRA, PRB...Program.
Claims
1. An acquisition unit acquires first information indicating at least the positional relationship in space between a first moving body and a second moving body whose position can change in accordance with the change in the position of the first moving body, and second information for tracking the position of the first moving body in space. A tracking unit that tracks the position of the first moving body in the space based on the second information, An estimation unit estimates the position of the second moving body in space based on the tracking result of the tracking unit and the first information, A motion tracking device equipped with the following features.
2. The system includes a detection unit that detects a first state in which the first moving body comes into contact with the second moving body based on the first information, The estimation unit estimates the position of the second moving body in space in the first state based on the tracking result of the tracking unit for the position of the first moving body. The motion tracking device according to claim 1.
3. The second moving body is connected to a third moving body, the third moving body having a fixed relative position to the second moving body and in contact with the first moving body in the first state. The motion tracking device according to claim 2, wherein the estimation unit tracks the position of the third motion body in the first state based on the position tracking result of the first motion body, and estimates the position of the second motion body in the first state based on the position tracking result of the third motion body and the relative position of the third motion body with respect to the second motion body.
4. The first moving body is connected to a fourth moving body whose relative position to the first moving body is known, and which the second moving body contacts in the first state. The motion tracking device according to claim 2, wherein the estimation unit tracks the position of the fourth motion body in the first state based on the position tracking result for the first motion body and the relative position of the fourth motion body with respect to the first motion body, and estimates the position of the second motion body in the first state based on the position tracking result for the fourth motion body.
5. The space includes a fifth moving body whose position changes in accordance with the change in the position of the first moving body, while its relative position to the second moving body is fixed. The motion tracking device according to claim 1, wherein the estimation unit tracks the position of the fifth motion body based on the position tracking result for the first motion body, and estimates the position of the second motion body based on the position tracking result for the fifth motion body and the relative position of the fifth motion body with respect to the second motion body.
6. An acquisition unit acquires, for a first moving body and a second moving body whose posture can change in accordance with the change in posture of the first moving body, first information showing the relationship between the posture of the first moving body and the posture of the second moving body in space, and second information for tracking the posture of the first moving body in space. A tracking unit that tracks the attitude of the first moving body based on the second information, An estimation unit estimates the posture of the second moving body based on the tracking result of the tracking unit and the first information, A motion tracking device equipped with the following features.
7. Computers, An acquisition unit acquires first information indicating at least the positional relationship in space between a first moving body and a second moving body whose position can change in accordance with the change in the position of the first moving body, and second information for tracking the position of the first moving body in space. A tracking unit that tracks the position of the first moving body in the space based on the second information, An estimation unit estimates the position of the second moving body in space based on the tracking result of the tracking unit and the first information, A program that makes something work.
8. Computers, An acquisition unit acquires, for a first moving body and a second moving body whose posture can change in accordance with the change in posture of the first moving body, first information showing the relationship between the posture of the first moving body and the posture of the second moving body in space, and second information for tracking the posture of the first moving body in space. A tracking unit that tracks the attitude of the first moving body based on the second information, An estimation unit estimates the posture of the second moving body based on the tracking result of the tracking unit and the first information, A program that makes something work.