Object tracking system and 3D position reporting method

Abstract

To provide a 3D location reporting method and an object tracking system for maximizing a reporting rate of 3D coordinates of landmarks on objects.SOLUTION: A method includes the steps of: capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, the plurality of 2D image frames and the plurality of 3D depth image frames being interleaved, and a first 2D image frame of the plurality of 2D image frames being captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D location according to an N-th 2D image frame of the plurality of 2D image frames and an N-th 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D location according to the N-th 3D depth image frame and an (N+1)-th 2D image frame of the plurality of 2D image frames.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an object tracking system and a 3D position reporting method, and more particularly to an object tracking system and a 3D position reporting method that reports a 3D position upon receiving each frame image. [Background technology]

[0002] As computer 3D vision applications become more mature and human-computer interaction sensing technologies advance with the metaverse trend, a smooth visual experience requires high-frame-rate tracking for applications such as virtual reality, augmented reality, and 3D naked-eye vision. These applications need to track features such as face, eyes, gestures, and pose by using RGB or grayscale images and depth images to obtain three-dimensional (3D) coordinates of landmarks on objects in the real world.

[0003] In this context, one of the industry goals is how to maximize the reporting rate of 3D landmark coordinates. Summary of the Invention

[0004] The present invention provides an object tracking system and a 3D position reporting method to solve the above problems.

[0005] An embodiment of the present invention provides an object tracking system, the object tracking system including: a sensor configured to capture a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; a processor coupled to the sensor; and a memory coupled to the processor and configured to store program code for instructing the processor to perform a 3D position reporting method, the 3D position reporting method including: calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames.

[0006] An embodiment of the present invention provides a 3D position reporting method, the 3D position reporting method including: capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames.

[0007] These and other objectives of the present invention will no doubt become obvious to those skilled in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of an object tracking system according to an embodiment of the present invention; [Figure 2] 1 is a process flowchart of a 3D position reporting method according to an embodiment of the present invention. [Figure 3] FIG. 2 is a timing diagram for performing a 3D position reporting method according to one embodiment of the present invention. [Figure 4A] 3D depth image frames of a human face tracked by an object tracking system according to one embodiment of the present invention. [Figure 4B] 1 is a portion of a 3D depth image frame around a human face according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Certain terms are used throughout the description and the following claims to refer to particular components. As one skilled in the art will appreciate, hardware manufacturers may refer to components by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and claims, the terms "include" and "comprise" are used in an open-ended manner and, therefore, should be interpreted to mean "including, but not limited to." Additionally, the term "couple" is intended to mean either an indirect or direct electrical connection. Thus, when a device is coupled to another device, the connection may be through a direct electrical connection or through an indirect electrical connection via other devices and connections.

[0010] Please refer to FIG. 1. FIG. 1 is a schematic diagram of an object tracking system 1 according to one embodiment of the present invention. The object tracking system 1 includes a sensor 10, a processor 20, and a memory 30. The sensor 10 and the memory 30 are coupled to the processor 20, which is used to represent the basic components of the object tracking system 1. The sensor 10 captures a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having a sensor frame rate. It should be noted that the plurality of 2D image frames may be, but are not limited to, near-infrared (NIR) image frames, RGB image frames, grayscale image frames, or the like. The memory 30 stores program code for instructing the processor 20 to perform a 3D position reporting method such that the processor 20 generates 3D positions of landmarks of the object and maximizes the 3D position reporting rate.

[0011] The 3D position reporting method of the object tracking system 1 can be summarized as Process 2, as shown in Figure 2. Process 2 includes the following steps:

[0012] Start at step S200.

[0013] In step S202, a first 3D position is calculated according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames.

[0014] In step S204, a second 3D position is calculated according to the Nth 3D depth image frame and the (N+1)th 2D image frame of the plurality of 2D image frames.

[0015] The process ends in step S206.

[0016] Please refer to FIG. 3. FIG. 3 is a timing diagram for performing a 3D position reporting method according to one embodiment of the present invention. In one embodiment, the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames. Specifically, the sensor 10 may capture one 2D image frame and one 3D depth image frame within each frame period corresponding to the sensor frame rate. For example, the sensor 10 captures an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames within the Nth frame period.

[0017] According to process 2, in step S202, the processor 20 calculates a first 3D position according to the Nth 2D image frame and the Nth 3D depth image frame. Specifically, as shown in FIG. 3 , the processor 20 calculates an Nth 2D position corresponding to the Nth 2D image frame, and calculates a first 3D position according to the Nth 2D position and the Nth 3D depth image frame. In step S204, the processor 20 calculates a second 3D position according to the Nth 3D depth image frame corresponding to the Nth 2D image frame and the (N+1)th 2D position corresponding to the (N+1)th 2D image frame. Specifically, the processor 20 stores the Nth 3D depth image frame calculated within the Nth frame period in the memory 30, and reuses the Nth 3D depth image frame calculated within the Nth frame period to calculate a second 3D position within the (N+1)th frame period. Similarly, processor 20 may calculate one first 3D position and one second 3D position in each frame period corresponding to the sensor frame rate. In other words, processor 20 may generate two 3D positions (a first 3D position and a second 3D position) of the object's landmark in each frame period. In this way, processor 20 may report one 3D position every half frame period, which means that the reporting rate of processor 20 for 3D positions is twice the sensor frame rate.

[0018] It should be noted that the processor 20 may, but is not limited to, reuse a 3D depth image frame stored in any previous frame period to calculate the second 3D position in the current frame period.

[0019] As mentioned above, the 3D depth image frame needs to be maintained until the next frame period and used along with the 2D position. However, maintaining the complete 3D depth image frame requires a large memory 30 footprint and long access times. In the present invention, a sensor 10 with a high sensor frame rate is used to ensure that the movement range of each image frame is limited within a specific range. Therefore, the present invention can also optimize and retain a portion of the 3D depth image frame to calculate the 3D position of the landmark in the next frame period. See FIGS. 4A and 4B. FIG. 4A is a 3D depth image frame of a human face tracked by an object tracking system 1 according to an embodiment of the present invention. FIG. 4B is a portion of a 3D depth image frame around the human face according to an embodiment of the present invention. Note that the object tracking system 1 may track a human eye within the human face, but is not limited to this. In particular, as shown in FIG. 4B, the processor 20 can calculate a target 3D depth image frame corresponding to the region of the human face within the 3D depth image frame. For example, the target 3D depth image frame may be a rectangular region around the human eye, with the length and width of the rectangular region being h and w, respectively. Specifically, the processor 20 may calculate the second 3D position according to the target 3D depth image frame and the (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames. In this manner, the 3D position reporting method may reduce memory usage and access time by avoiding maintaining a complete 3D depth image frame in the memory 30.

[0020] Furthermore, the processor 20 may calculate the following formula:

number

[0021] In the formula, (X, Y) represent the coordinates of at least one landmark, (u, v) represent the coordinates of the start point of the region corresponding to the at least one landmark, (w, h) represent the width and length of the region, and ε w and ε h and represent the upper limit of the moving distance of an object between two consecutive frame periods in the horizontal and vertical directions, respectively.

[0022] For example, as shown in Figures 4A and 4B, the processor 20 determines the human eye as two landmarks (x1, y1) and (x2, y2). According to these equations, the processor 20 obtains the region corresponding to the landmarks (x1, y1) and (x2, y2). In this manner, the processor 20 may calculate a second 3D position according to the target 3D depth image frame corresponding to the region and the 2D position of (N+1). In this manner, the 3D position reporting method may reduce memory usage and access time by avoiding maintaining a complete 3D depth image frame in the memory 30.

[0023] It should be noted that the processor 20 may obtain multiple regions corresponding to multiple landmarks and is not limited to one region, for example, the processor 20 may obtain two regions each corresponding to a human eye.

[0024] Those skilled in the art will readily combine, modify, and / or alter the above-described descriptions and examples. The above-described descriptions, steps, procedures, and / or processes, including suggested steps, may be implemented by means that may be hardware, software, firmware (known as a combination of hardware devices and computer instructions and data residing as read-only software on the hardware devices), electronic systems, or combinations thereof. Examples of hardware may include analog, digital, and mixed circuits known as microcircuits, microchips, or silicon chips. Examples of electronic systems may include systems-on-chips (SoCs), systems-in-packages (SiPs), computer-on-modules (CoMs), and air leak detection systems. Any of the above-described procedures and examples may be compiled into program code or instructions stored in memory 30. Memory 30 may include, but is not limited to, read-only memory (ROM), flash memory, random-access memory (RAM), subscriber identity module (SIM), hard disk, CD-ROM / DVD-ROM / BD-ROM, etc. The processor 20 may read and execute program codes or instructions stored in the memory 30 to implement the functions described above.

[0025] In summary, the prior art can report only one 3D position within each frame period. In comparison, the 3D position reporting method of the present invention can report two 3D positions within each frame period. In other words, the reporting rate of the present invention is twice that of the prior art. In this way, the present invention maximizes the reporting rate of 3D positions to achieve a smooth viewing experience.

[0026] Those skilled in the art will readily appreciate that numerous modifications and variations of the devices and methods may be made while retaining the teachings of the present invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.

Claims

1. a sensor configured to capture a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; a processor coupled to the sensor; a memory coupled to the processor and configured to store program code for instructing the processor to perform a 3D position reporting method; 1. An object tracking system comprising: calculating a first 3D position of a landmark on an object according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; calculating a second 3D position of the landmark according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames; Including, an object tracking system.

2. The object tracking system of claim 1 , wherein the plurality of 2D image frames are near-infrared (NIR) image frames or RGB image frames.

3. The step of calculating the first 3D position comprises: calculating a plurality of 2D positions of the landmark corresponding to the plurality of 2D image frames respectively; calculating the first 3D position according to an Nth 2D position of the landmark corresponding to the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames; The object tracking system of claim 1 , comprising:

4. The step of calculating the second 3D position comprises: calculating the second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D position of the landmark corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; The object tracking system of claim 3 further comprising:

5. The step of calculating the second 3D position comprises: calculating a target 3D depth image frame corresponding to a region of the object in the Nth 3D depth image frame; calculating the second 3D position according to the target 3D depth image frame and an (N+1)th 2D position of the landmark corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; The object tracking system of claim 3 further comprising:

6. capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D position of a landmark on an object according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; calculating a second 3D position of the landmark according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames; 3D position reporting method, comprising:

7. The 3D position reporting method of claim 6 , wherein the plurality of 2D image frames are near-infrared (NIR) image frames or RGB image frames.

8. The step of calculating the first 3D position comprises: calculating a plurality of 2D positions of the landmark corresponding to the plurality of 2D image frames respectively; calculating the first 3D position according to an Nth 2D position of the landmark corresponding to the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames; 7. The 3D position reporting method of claim 6, comprising:

9. The step of calculating the second 3D position comprises: calculating the second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D position of the landmark corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; The 3D position reporting method of claim 8 further comprising:

10. The step of calculating the second 3D position comprises: calculating a target 3D depth image frame corresponding to a region of the object in the Nth 3D depth image frame; calculating the second 3D position according to the target 3D depth image frame and an (N+1)th 2D position of the landmark corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; The 3D position reporting method of claim 8 further comprising:

Images