Human body part tracking method and human body part tracking system

Abstract

A human body part tracking method and a human body part tracking system are provided. A first image is obtained. The first image captures a first segment and a second segment of a human body part at a first time point. A first reference point and a second reference point are identified from the first image. A positional relationship between the first segment and the second segment is determined according to three-dimensional coordinates of the first reference point and the second reference point. A second image is obtained. The second image captures the first segment but not the second segment at a second time point. A third reference point is identified from the second image. The third reference point is identified from the second image. A three-dimensional coordinate of a fourth reference point is predicted by using the three-dimensional coordinate of the third reference point and the positional relationship. The fourth reference point indicates a position of the second segment of the human body part at the second time point. Thus, the position of the human body part outside the field of view can be tracked.

Description

Technical Field

[0001] This disclosure generally relates to motion prediction, and more specifically, to methods and systems for tracking parts of the human body. Background Technology

[0002] Extended reality (XR) technologies, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), are currently popular for simulating sensations, perceptions, and / or environment. These technologies can be applied in various fields, including gaming, military training, healthcare, and remote work. Typically, users wear head-mounted displays to experience virtual worlds. Furthermore, to provide intuitive operation on head-mounted display systems, user movement can be detected to directly operate the system based on that movement. User movement can be determined based on one or more images captured by a camera. However, cameras have limitations in their field of view. For example, Figure 1A and Figure 1B This is a schematic diagram illustrating an example of hand movement. (Reference) Figure 1A The user's hand (H) is within the camera's field of view (FOV). (Reference) Figure 1B When the user raises their hand H further, the hand H may be outside the field of view (FOV). The head-mounted display system may not be aware of this. Figure 1B The position of hand H in the image cannot be tracked, and the movement of hand H cannot be continued. Summary of the Invention

[0003] When a part of the human body is outside the field of view, its position may not be trackable. Therefore, this disclosure relates to a method and system for tracking human body parts to predict the position of missing parts of the human body within the camera's field of view.

[0004] In one exemplary embodiment, the human body segment tracking method includes (but is not limited to) the following steps: Obtaining a first image from an image capture device, wherein the first image captures a first segment and a second segment of a human body at a first time point. Identifying a first reference point and a second reference point from the first image. The first reference point indicates the position of the first segment at the first time point, and the second reference point indicates the position of the second segment at the first time point. Determining a positioning relationship between the first segment and the second segment based on the three-dimensional coordinates of the first and second reference points. Obtaining a second image from the image capture device. The second image captures the first segment but not the second segment at a second time point. Identifying a third reference point from the second image. Identifying the third reference point from the second image, wherein the third reference point indicates the position of the first segment of the human body at the second time point. Predicting the three-dimensional coordinates of a fourth reference point using the three-dimensional coordinates of the third reference point and the positioning relationship. The fourth reference point indicates the position of the second segment of the human body at the second time point.

[0005] In one exemplary embodiment, the human body part tracking system includes (but is not limited to) an image capture device and a processor. The processor is coupled to the image capture device and configured to: acquire a first image via the image capture device, wherein the first image captures a first segment and a second segment of a human body part at a first time point, and the first segment of the human body part is connected to the second segment of the human body part; identify a first reference point and a second reference point from the first image, wherein the first reference point indicates the position of the first segment of the human body part at the first time point, and the second reference point indicates the position of the second segment of the human body part at the first time point; determine a positioning relationship between the first segment and the second segment of the human body part based on the three-dimensional coordinates of the first and second reference points; acquire a second image via the image capture device, wherein the second image captures the first segment of the human body part at a second time point but not the second segment; identify a third reference point from the second image, wherein the third reference point indicates the position of the first segment of the human body part at the second time point; predict the three-dimensional coordinates of a fourth reference point using the three-dimensional coordinates of the third reference point and the positioning relationship, wherein the fourth reference point indicates the position of the second segment of the human body part at the second time point; and determine the positioning relationship between the first and second segments based on the first image.

[0006] Based on the above, according to the human body part tracking method and system of the present invention, the positions of reference points of two segments of a human body part in an image at a first time point can be used to predict the position of a reference point of a segment in another image that is not located at a second time point. Therefore, even when a human body part is hidden by something, the position of the hidden part can still be tracked.

[0007] However, it should be understood that this invention may not contain all aspects and embodiments of this disclosure, is not intended to be limiting or restrictive in any way, and the invention disclosed herein is to be understood by and will be understood by those skilled in the art to cover obvious improvements and modifications thereto. Attached Figure Description

[0008] The accompanying drawings are included to provide a further understanding of this disclosure and are incorporated in and form a part of this specification. The drawings illustrate embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0009] Figure 1A and Figure 1B This is a schematic diagram illustrating an example of hand movement;

[0010] Figure 2 This is a block diagram illustrating a human body part tracking system according to one of the exemplary embodiments of the present disclosure;

[0011] Figure 3 This is a flowchart illustrating a human body part tracking method according to one of the exemplary embodiments of the present disclosure;

[0012] Figure 4 This is a schematic diagram illustrating the movement of a human body part at a first time point according to an exemplary embodiment of the present disclosure;

[0013] Figure 5 This is a schematic diagram illustrating the movement of a human body part at a second time point according to an exemplary embodiment of the present disclosure;

[0014] Figure 6 This is a schematic diagram illustrating the movement of a human body portion at a second time point according to an exemplary embodiment of the present disclosure.

[0015] Explanation of icon numbers

[0016] 100: Human body part tracking system;

[0017] 110: Image capturing device;

[0018] 130: Memory;

[0019] 150: Processor;

[0020] 410: Hands;

[0021] 411: Second reference point;

[0022] 413: Fourth reference point;

[0023] 415: Target point;

[0024] 430: Forearm;

[0025] 431: First reference point;

[0026] 435: Base point;

[0027] 433, 453: Third reference points;

[0028] 451: Intersection point;

[0029] 455: Midpoint;

[0030] S310, S320, S330, S340, S350, S360: Steps;

[0031] BL: Physical Connection;

[0032] FOV: Field of View;

[0033] H: Hand. Detailed Implementation

[0034] Reference will now be made in detail to the present preferred embodiments of this disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and description to refer to the same or similar parts.

[0035] Figure 2 This is a block diagram illustrating a human body part tracking system 100 according to one exemplary embodiment of the present disclosure. Reference Figure 2 The human body tracking system 100 includes (but is not limited to) an image capture device 110, a memory 130, and a processor 150. The human body tracking system 100 is suitable for XR (e.g., VR, AR, MR, or other reality simulation-related technologies).

[0036] Image capture device 110 may be a camera, such as a monochrome or color camera, a depth camera, a video recorder, or other image sensor capable of capturing images. In one embodiment, image capture device 110 is mounted on the body of a head-mounted display (HMD) and oriented in a specific direction for capture. For example, when a user wears the HMD, image capture device 110 captures the scene in front of the user. In some embodiments, the orientation and / or field of view of image capture device 110 may be adjusted as needed. In yet other embodiments, image capture device 110 may be used to capture images of one or more parts of the user's body to generate images containing those parts. For example, one or more body parts may include the user's hands, arms, ankles, legs, or other body parts.

[0037] Memory 130 can be any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, similar devices, or combinations thereof. Memory 130 records program code, device configuration, buffered data, or permanent data (such as images, positions, positioning relationships, three-dimensional coordinates, and motion models) and will be introduced later.

[0038] Processor 150 is coupled to image capture device 110 and memory 130. Processor 150 is configured to load program code stored in memory 130 to perform the processes of exemplary embodiments of this disclosure.

[0039] In some embodiments, processor 150 may be a central processing unit (CPU), a microprocessor, a microcontroller, a digital signal processing (DSP) chip, or a field-programmable gate array (FPGA). The functionality of processor 150 may also be implemented by a separate electronic device or an integrated circuit (IC), and the operation of processor 150 may also be implemented by software.

[0040] It should be noted that the processor 150 may not be housed on the same device as the image capture device 110. However, the devices are each equipped with the image capture device 110, and the processor 150 may also include a communication transceiver or physical transmission line with compatible communication technologies (such as Bluetooth, Wi-Fi, and IR wireless communication) to transmit or receive data with each other. For example, the processor 150 may be housed in a computing device, while the image capture device 110 may be housed on the main body of the HMD.

[0041] To better understand the operation provided in one or more embodiments of this disclosure, several embodiments will be illustrated below to explain in detail the operation of the human body tracking system 100. The devices and modules in the human body tracking system 100 are applied in the following embodiments to explain the control methods provided herein. Each step of the method may be adjusted according to actual implementation and should not be limited to what is described herein.

[0042] Figure 3 This is a flowchart illustrating a human body part tracking method according to one exemplary embodiment of the present disclosure. Reference Figure 3The processor 150 can obtain a first image from / from the image capture device 110 (step S310). Specifically, the first image is one of the images captured by the image capture device 110 at a first time point. It should be noted that the first image captures a first segment and a second segment of a human body at the first time point. That is, both the first and second segments of the human body are within the field of view of the image capture device 110. The human body can be a user's hand, arm, leg, foot, ankle, or other human body part. The first segment of the human body is connected to the second segment of the human body. The first and second segments can be parts of the human body between joints or any two adjacent parts of the human body. For example, the first segment is a forearm and the second segment is a hand. Or, for another example, the first segment is a palm and the second segment is a finger. In some embodiments, these segments are determined based on actual needs. The processor 150 can further identify the human body part, the first segment, and the second segment in the first image using machine learning techniques configured with object recognition capabilities (such as deep learning, artificial neural networks (ANN), or support vector machines (SVM), etc.) or other image recognition techniques.

[0043] Processor 150 can identify a first reference point and a second reference point from the first image (step S320). In one embodiment, the first reference point indicates the position of a first segment of the human body at a first time point, and the second reference point indicates the position of a second segment of the human body at the first time point. The first and second reference points can be centroids, geometric centers, or any points located at the first and second segments, respectively. In some embodiments, additional reference points, surfaces, or contours can be used to indicate the position of the first or second segment.

[0044] Figure 4 This is a schematic diagram illustrating the movement of a human body portion at a first time point according to an exemplary embodiment of the present disclosure. (Reference) Figure 4 At the first time point, both the hand 410 and the forearm 430 are within the field of view (FOV). Furthermore, the processor 150 determines a point located at the forearm 430 and the center of gravity of the hand 410 as the first reference point 431 and the second reference point 411, respectively.

[0045] The processor 150 can determine the positioning relationship between the first and second segments of the human body part based on the three-dimensional coordinates of the first and second reference points (step S330). Specifically, three-dimensional coordinates are one format for representing the position of a point in space. The sensing intensity and pixel positioning corresponding to the first and second reference points in the first image can be used to estimate the depth information of the first and second reference points (i.e., the distance relative to the image capturing device 110 or other reference devices) and estimate the two-dimensional coordinates of the first and second reference points in a plane parallel to the image capturing device 110, so as to generate the three-dimensional coordinates of the first and second reference points. In some embodiments, relative position or depth can be used to represent the position of a point.

[0046] Furthermore, the positioning relationship involves the relative positioning between the first and second segments of the human body. For example, the positioning relationship involves the distance between the first and second segments and / or the direction from the first segment to the second segment.

[0047] In one embodiment, the processor 150 may connect a first reference point of a first segment and a second reference point of a second segment in a first image to form a body connection between the first reference point and the second reference point as a positioning relationship. Figure 4 As an example, the body connection BL connects the first reference point 431 and the second reference point 411. That is, the first reference point 431 and the second reference point 411 are the two ends of the body connection BL.

[0048] In some embodiments, the body connection BL may not be a straight line. An intersection point 451 may be generated between the hand 410 and the forearm 430. For example, the intersection point 451 is located at the wrist. Furthermore, the body connection BL may extend further through the intersection point 451.

[0049] Processor 150 can acquire a second image from / from image capture device 110 (step S340). Specifically, the second image is another image among images captured by image capture device 110 at a second time point after the first time point. It should be noted that the second image captures the first segment but not the second segment at the second time point. That is, the human body part moves and only the first segment of the human body part is within the field of view of image capture device 110.

[0050] Processor 150 may identify a third reference point from the second image (step S350). In one embodiment, the third reference point indicates the position of a first segment of the human body at a second time point. The third reference point may be the center of gravity, geometric center, or any point located at the first segment. In some embodiments, further reference points, surfaces, or contours may be used to indicate the position of the first segment at the second time point.

[0051] Figure 5This is a schematic diagram illustrating the movement of a human body portion at a second time point according to an exemplary embodiment of the present disclosure. (Reference) Figure 5 At the second time point, only the forearm 430 is within the field of view (FOV), while the hand 410 is outside the FOV. Furthermore, the processor 150 designates a point located at the forearm 430 as a third reference point 433.

[0052] The processor can predict the three-dimensional coordinates of the fourth reference point using the three-dimensional coordinates and positioning relationships of the third reference point (step S360). In one embodiment, the fourth reference point indicates the position of the second segment of the human body at the second time point. The fourth reference point can be the center of gravity, geometric center, or any point located at the second segment. In some embodiments, further reference points, surfaces, or contours can be used to indicate the position of the second segment at the second time point.

[0053] Because the fourth reference point may not be within the field of view, the processor 150 can directly determine the position of the second segment as the position of the first segment without relying on the second image, assuming the positioning relationship involves the positions of the third and fourth reference points. The relative positioning between the first and second segments at the first time point can also be the same as the relative positioning between the first and second segments at the second time point.

[0054] In one embodiment, processor 150 can determine the three-dimensional coordinates of the fourth reference point by connecting the third and fourth reference points in the second image together with the body connection. The body connection can retain its shape. However, the two ends of the body connection will change from the first and second reference points to the third and fourth reference points. Processor 150 can determine the coordinate difference between the first and second reference points based on the body connection, and use the three-dimensional coordinates of the third reference point and the coordinate difference to determine the three-dimensional coordinates of the fourth reference point.

[0055] use Figure 4 and Figure 5 As an example, the body connection BL can shift with the movement of the hand 410 and forearm 430. A body connection BL exists connecting a third reference point 433 (third reference point 453) and a fourth reference point 413. Therefore, the position of the fourth reference point 413 can be determined.

[0056] In one embodiment, processor 150 can determine a base point located at the end of the first segment but not at the second segment. For example, the first segment is the forearm, the second segment is the hand, and the base point is located at the elbow joint. Based on inverse kinematics, processor 150 can estimate the position of a target point located at the second segment based on the positions of the midpoint and the base point located between the first and second segments in the second image. For example, using the elbow joint as the base point, the midpoint could be located at the wrist, which is the intersection of the forearm and the hand, and the target point could be the tip of a finger. On the other hand, inverse kinematics is a mathematical procedure for calculating the parameters of the junction point. Processor 150 can consider the midpoint, the base point, and the target point as junction points for inverse kinematics. Based on inverse kinematics, the position of the target point can be estimated given junction point parameters (e.g., the angle between the first and second segments, the positions of the base point and the midpoint, etc.). Subsequently, processor 150 can adjust the three-dimensional coordinates of a fourth reference point based on the position of the target point. Assume that both the fourth reference point and the target point are located at the second segment of the human body. The processor 150 can use the target point to check whether the fourth reference point has deviated from the correction position and further modify the three-dimensional coordinates of the fourth reference point.

[0057] Figure 6 This is a schematic diagram illustrating the movement of a human body portion at a second time point according to an exemplary embodiment of the present disclosure. (Reference) Figure 6 The base point 435 is located at the elbow joint, the midpoint 455 is located at the wrist, and the target point 415 is located at the fingertip. The position of the target point 415 can be determined based on the base point 435 and the midpoint 455 using inverse dynamics. The processor 150 can determine whether the fourth reference point is located on the line connecting the midpoint 455 and the target point 415. The processor 150 can further modify the three-dimensional coordinates of the fourth reference point 413 based on the line connecting the midpoint 455 and the target point 415.

[0058] In one embodiment, processor 150 may determine a motion model of the second segment based on a first image and one or more previous images. One or more previous images are acquired from image capture device 110 prior to the first image. Processor 150 may analyze the displacement of the second segment in the first image and the previous images, and further estimate the trajectory, rotation, and / or velocity of the second segment. Alternatively, the motion model is a mathematical model simulating the motion of the second segment. The trajectory, rotation, and / or velocity of the second segment can be used to estimate the motion model. For example, if the velocity remains constant, the motion model may be constant velocity motion. Processor 150 may further adjust the three-dimensional coordinates of a fourth reference point based on the motion model. For example, if the motion model is rotational motion, the position of the fourth reference point may be lowered.

[0059] In one embodiment, processor 150 can adjust the three-dimensional coordinates of the fourth reference point based on a motion model according to the position of the target point. That is, processor 150 can further adjust the modified three-dimensional coordinates of the fourth reference point using the position of the target point based on the motion model. For example, processor 150 determines whether the modified three-dimensional coordinates of the fourth reference point lie on the line connecting the midpoint and the target point. Therefore, the accuracy of position estimation can be improved.

[0060] In summary, in the human body part tracking method and system of this invention, the positional relationship between two segments of a human body part in an image at a first time point can be determined based on two reference points. If a segment is outside the field of view of the image capture device at a second time point, the position of the reference point corresponding to the vanished segment can be estimated based on the positional relationship. Therefore, even when a portion of the human body part disappears from the field of view, that portion can still be tracked.

[0061] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of this disclosure without departing from its scope or spirit. In view of the foregoing, it is intended that this disclosure cover such modifications and variations, provided that such modifications and variations fall within the scope of the following claims and their equivalents.

Claims

1. A human body part tracking method, comprising: obtaining a first image from an image capturing device, wherein the first image captures a first segment and a second segment of a human body part at a first time point, and the first segment of the human body part is connected to the second segment of the human body part; identifying a first reference point and a second reference point from the first image, wherein the first reference point indicates a position of the first segment of the human body part at the first time point, and the second reference point indicates a position of the second segment of the human body part at the first time point; determining a positional relationship between the first segment and the second segment of the human body part according to three-dimensional coordinates of the first reference point and the second reference point; obtaining a second image from the image capturing device, wherein the second image captures the first segment but not the second segment of the human body part at a second time point; identifying a third reference point from the second image, wherein the third reference point indicates a position of the first segment of the human body part at the second time point; and predicting three-dimensional coordinates of a fourth reference point by using the three-dimensional coordinates of the third reference point and the positional relationship, wherein the fourth reference point indicates a position of the second segment of the human body part at the second time point.

2. The human body part tracking method of claim 1, wherein the step of determining the positional relationship between the first segment and the second segment comprises: connecting the first reference point of the first segment and the second reference point of the second segment in the first image to form a body connection between the first reference point and the second reference point as the positional relationship.

3. The human body part tracking method of claim 2, wherein the step of predicting the three-dimensional coordinates of the fourth reference point comprises: determining the three-dimensional coordinates of the fourth reference point by connecting the third reference point and the fourth reference point with the body connection in the second image.

4. The human body part tracking method of claim 3, further comprising: determining a base point at an end of the first segment at the second time point; and estimating a position of a target point at the second segment based on positions of a mid-point between the first segment and the second segment in the second image and the base point according to inverse dynamics, wherein the inverse dynamics is a mathematical procedure that calculates parameters of a joint, and the mid-point, the base point, and the target point are considered as the joint; and adjusting the three-dimensional coordinates of the fourth reference point according to the position of the target point.

5. The human body part tracking method of claim 3, further comprising: determining a motion model of the second segment according to the first image and at least one previous image, wherein the at least one previous image is obtained before the first image, and the motion model is a mathematical model that simulates a motion of the second segment; and adjusting the three-dimensional coordinates of the fourth reference point according to the motion model.

6. The human body part tracking method of claim 5, further comprising: determine a base point located at an end of the first segment at the second time point; and estimate a position of a target point located at the second segment based on positions of the base point and a midpoint between the first segment and the second segment in the second image according to inverse dynamics, wherein the inverse dynamics is a mathematical procedure that calculates parameters of a joint, and the base point, the midpoint, and the target point are considered as the joint; and adjust the three-dimensional coordinates of the fourth reference point based on the motion model according to the position of the target point.

7. A human body part tracking system, comprising: an image capturing device; a processor coupled to the image capturing device and configured to: obtain a first image by the image capturing device, wherein the first image captures a first segment and a second segment of a human body part at a first time point, and the first segment of the human body part is connected to the second segment of the human body part; identify a first reference point and a second reference point from the first image, wherein the first reference point indicates a position of the first segment of the human body part at the first time point, and the second reference point indicates a position of the second segment of the human body part at the first time point; determine a positional relationship between the first segment and the second segment of the human body part according to three-dimensional coordinates of the first reference point and the second reference point; obtain a second image by the image capturing device, wherein the second image captures the first segment of the human body part but not the second segment at a second time point; identify a third reference point from the second image, wherein the third reference point indicates a position of the first segment of the human body part at the second time point; and predict three-dimensional coordinates of a fourth reference point by using the three-dimensional coordinates of the third reference point and the positional relationship, wherein the fourth reference point indicates a position of the second segment of the human body part at the second time point.

8. The human body part tracking system of claim 7, wherein the processor is configured to: connect the first reference point of the first segment and the second reference point of the second segment in the first image to form a body connection between the first reference point and the second reference point as the positional relationship.

9. The human body part tracking system of claim 8, wherein the processor is configured to: determine the three-dimensional coordinates of the fourth reference point by connecting the third reference point and the fourth reference point in the second image together with the body connection.

10. The human body part tracking system of claim 9, wherein the processor is configured to: determine a base point located at an end of the first segment at the second time point; and estimate a position of a target point located at the second segment based on positions of the base point and a midpoint between the first segment and the second segment in the second image according to inverse dynamics, wherein the inverse dynamics is a mathematical procedure that calculates parameters of a joint, and the midpoint, the base point, and the target point are considered as the joint; and adjusting the three-dimensional coordinates of the fourth reference point according to the position of the target point.

11. The body part tracking system of claim 9, wherein the processor is configured to: determine a motion model of the second segment from the first image and at least one previous image, wherein the at least one previous image is obtained prior to the first image, and the motion model is a mathematical model that models motion of the second segment; and adjust the three-dimensional coordinates of the fourth reference point according to the motion model.

12. The body part tracking system of claim 11, wherein the processor is configured to: determine a base point that is located at an end of the first segment at the second time point; and estimate a position of a target point that is located at the second segment according to positions of a mid-point between the first segment and the second segment in the second image and the base point based on inverse dynamics, wherein the inverse dynamics is a mathematical procedure that calculates parameters of a joint, and the mid-point, the target point, and the base point are considered as the joint; and adjust the three-dimensional coordinates of the fourth reference point according to the position of the target point based on the motion model.

13. The body part tracking system of claim 12, wherein the processor is configured to: determine a first motion model of the first segment from the first image and at least one previous image, wherein the at least one previous image is obtained prior to the first image, and the first motion model is a mathematical model that models motion of the first segment; and adjust the three-dimensional coordinates of the fourth reference point according to the first motion model.

14. The body part tracking system of claim 13, wherein the processor is configured to: determine a second motion model of the second segment from the second image and at least one previous image, wherein the at least one previous image is obtained prior to the second image, and the second motion model is a mathematical model that models motion of the second segment; and adjust the three-dimensional coordinates of the fourth reference point according to the second motion model.

Images