Object tracking system and 3D location reporting method

Abstract

To provide a 3D location reporting method and an object tracking system for maximizing a reporting rate of 3D coordinates of landmarks on objects.SOLUTION: A method includes the steps of: capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, the plurality of 2D image frames and the plurality of 3D depth image frames being interleaved, and a first 2D image frame of the plurality of 2D image frames being captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D location according to an N-th 2D image frame of the plurality of 2D image frames and an N-th 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D location according to the N-th 3D depth image frame and an (N+1)-th 2D image frame of the plurality of 2D image frames.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an object tracking system and a 3D position reporting method, and more particularly to an object tracking system and a 3D position reporting method for reporting a 3D position upon receiving each frame image. [Background technology]

[0002] As computer 3D vision applications become more mature and human-computer interaction sensing technologies advance with the metaverse trend, a smooth visual experience requires high frame rate tracking for applications such as virtual reality, augmented reality, and 3D naked-eye vision. These applications need to track features such as face, eyes, gestures, and poses by using RGB or grayscale images and depth images to obtain three-dimensional (3D) coordinates of landmarks on objects in the real world.

[0003] In this context, one of the goals in the industry is how to maximize the reporting rate of 3D landmark coordinates. Summary of the Invention

[0004] The present invention provides an object tracking system and a 3D position reporting method to solve the above problems.

[0005] An embodiment of the present invention provides an object tracking system, comprising: a sensor configured to capture a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; a processor coupled to the sensor; and a memory coupled to the processor and configured to store program code for instructing the processor to perform a 3D position reporting method, the 3D position reporting method comprising: calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames.

[0006] An embodiment of the present invention provides a 3D position reporting method, the 3D position reporting method including: capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames.

[0007] These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings. [Brief description of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of an object tracking system according to an embodiment of the present invention; [Diagram 2] 4 is a process flowchart of a 3D position reporting method according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a timing diagram of performing a 3D position reporting method according to an embodiment of the present invention. [Figure 4A] 4 is a 3D depth image frame of a human face tracked by an object tracking system according to one embodiment of the present invention. [Figure 4B] 1 is a portion of a 3D depth image frame around a human face according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Certain terms are used throughout the description and the following claims to refer to certain components. As one skilled in the art will appreciate, hardware manufacturers may refer to components by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and claims, the terms "include" and "comprise" are used in a non-limiting manner and should therefore be interpreted to mean "including, but not limited to." Additionally, the term "couple" is intended to mean either an indirect or direct electrical connection. Thus, when a device is coupled to another device, the connection may be through a direct electrical connection or through an indirect electrical connection via other devices and connections.

[0010] Please refer to FIG. 1. FIG. 1 is a schematic diagram of an object tracking system 1 according to an embodiment of the present invention. The object tracking system 1 includes a sensor 10, a processor 20, and a memory 30. The sensor 10 and the memory 30 are coupled to the processor 20, which is used to represent the basic components of the object tracking system 1. The sensor 10 captures a plurality of 2D image frames having a sensor frame rate, and a plurality of 3D depth image frames having a sensor frame rate. It is noted that the plurality of 2D image frames may be, but are not limited to, near infrared (NIR) image frames, RGB image frames, or grayscale image frames, etc. The memory 30 stores program code for instructing the processor 20 to execute a 3D position reporting method such that the processor 20 generates 3D positions of landmarks of the object and maximizes the reporting rate of the 3D positions.

[0011] The 3D position reporting method of the object tracking system 1 can be summarized as Process 2, as shown in Figure 2. Process 2 includes the following steps:

[0012] Start at step S200.

[0013] In step S202, a first 3D position is calculated according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames.

[0014] In step S204, a second 3D position is calculated according to the Nth 3D depth image frame and the (N+1)th 2D image frame of the plurality of 2D image frames.

[0015] The process ends in step S206.

[0016] Please refer to Fig. 3. Fig. 3 is a timing diagram of performing a 3D position reporting method according to an embodiment of the present invention. In one embodiment, the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames. Specifically, the sensor 10 can capture one 2D image frame and one 3D depth image frame within each frame period corresponding to the sensor frame rate. For example, the sensor 10 captures an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames within an Nth frame period.

[0017] According to the process 2, in step S202, the processor 20 calculates a first 3D position according to the Nth 2D image frame and the Nth 3D depth image frame. In detail, as shown in FIG. 3, the processor 20 calculates an Nth 2D position corresponding to the Nth 2D image frame, and calculates a first 3D position according to the Nth 2D position and the Nth 3D depth image frame. In step S204, the processor 20 calculates a second 3D position according to the Nth 3D depth image frame corresponding to the Nth 2D image frame, and the (N+1)th 2D position corresponding to the (N+1)th 2D image frame. In detail, the processor 20 stores the Nth 3D depth image frame calculated within the Nth frame period in the memory 30, and reuses the Nth 3D depth image frame calculated within the Nth frame period to calculate a second 3D position within the (N+1)th frame period. Similarly, the processor 20 may calculate one first 3D position and one second 3D position in each frame period corresponding to the sensor frame rate. In other words, the processor 20 may generate two 3D positions (the first 3D position and the second position) of the landmark of the object in each frame period. In this way, the processor 20 may report one 3D position every half frame period, which means that the reporting rate of the processor 20 for the 3D positions is twice the sensor frame rate.

[0018] It should be noted that the processor 20 may, but is not limited to, reuse a 3D depth image frame stored in any previous frame period to calculate the second 3D position in the current frame period.

[0019] As mentioned above, the 3D depth image frame needs to be kept until the next frame period and used with the 2D position, but keeping the complete 3D depth image frame will result in high memory 30 usage and long access time. In the present invention, a sensor 10 with a high sensor frame rate is used to ensure that the movement range of each image frame is limited within a certain range. Therefore, the present invention can also optimize and retain a portion of the 3D depth image frame to calculate the 3D position of the landmark in the next frame period. Please refer to Figures 4A and 4B. Figure 4A is a 3D depth image frame of a human face tracked by the object tracking system 1 according to an embodiment of the present invention. Figure 4B is a portion of a 3D depth image frame around the human face according to an embodiment of the present invention. It should be noted that the object tracking system 1 may track the human eye within the human face, but is not limited thereto. In particular, as shown in Figure 4B, the processor 20 can calculate a target 3D depth image frame corresponding to the region of the human face in the 3D depth image frame. For example, a rectangular region around the human eye, the length and width of the rectangular region being h and w, respectively. Specifically, the processor 20 may calculate the second 3D position according to the target 3D depth image frame and the (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames. In this manner, the 3D position reporting method may reduce memory usage and access time by avoiding maintaining a complete 3D depth image frame in the memory 30.

[0020] Furthermore, the processor 20 may be configured to

number

[0021] In the formula, (X,Y) represents the coordinates of at least one landmark, (u,v) represents the coordinates of the start point of the region corresponding to the at least one landmark, (w,h) represents the width and length of the region, and ε w and ε h and represent the upper limit of the moving distance of an object between two consecutive frame periods in the horizontal and vertical directions, respectively.

[0022] For example, as shown in Figures 4A and 4B, the processor 20 determines the human eye as two landmarks (x1, y1) and (x2, y2). According to these formulas, the processor 20 obtains the area corresponding to the landmarks (x1, y1) and (x2, y2). In this way, the processor 20 may calculate the second 3D position according to the target 3D depth image frame corresponding to the area and the 2D position of (N+1). In this way, the 3D position reporting method may reduce memory usage and access time by avoiding maintaining a complete 3D depth image frame in the memory 30.

[0023] It should be noted that the processor 20 may obtain multiple regions corresponding to multiple landmarks and is not limited to one region. For example, the processor 20 may obtain two regions corresponding to human eyes, respectively.

[0024] Those skilled in the art should easily make combinations, modifications, and / or variations to the above description and examples. The above description, steps, procedures, and / or processes including the proposed steps may be realized by means that may be hardware, software, firmware (known as a combination of hardware devices and computer instructions and data residing as read-only software on the hardware devices), electronic systems, or combinations thereof. Examples of hardware may include analog, digital, and mixed circuits known as microcircuits, microchips, or silicon chips. Examples of electronic systems may include system-on-chip (SoC), system-in-package (SiP), computer-on-module (CoM), and air leak detection system 1. Any of the above procedures and examples may be compiled into program code or instructions stored in memory 30. Memory 30 may include, but is not limited to, read-only memory (ROM), flash memory, random access memory (RAM), subscriber identity module (SIM), hard disk, CD-ROM / DVD-ROM / BD-ROM, etc. The processor 20 may read and execute program codes or instructions stored in the memory 30 to implement the functions described above.

[0025] In summary, the prior art can only report one 3D position within each frame period. In comparison, the 3D position reporting method of the present invention can report two 3D positions within each frame period. In other words, the reporting rate of the present invention is twice that of the prior art. In this way, the present invention maximizes the reporting rate of 3D positions to achieve a smooth visual experience.

[0026] Those skilled in the art will readily appreciate that numerous modifications and variations of the devices and methods may be made while retaining the teachings of the present invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.

Claims

1. a sensor configured to capture a plurality of 2D image frames having a sensor frame rate, and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; a processor coupled to the sensor; a memory coupled to the processor and configured to store program code for instructing the processor to perform a 3D position reporting method; 1. An object tracking system, comprising: calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; calculating a second 3D position according to the N 3D depth image frame and an (N+1) 2D image frame of the plurality of 2D image frames; Includes an object tracking system.

2. The object tracking system of claim 1 , wherein the plurality of 2D image frames are near-infrared (NIR) image frames or RGB image frames.

3. The step of calculating the first 3D position comprises: calculating a plurality of 2D positions corresponding respectively to the plurality of 2D image frames; calculating the first 3D position according to an Nth 2D position corresponding to the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames; The object tracking system of claim 1 , comprising:

4. The step of calculating the second 3D position comprises: calculating the second 3D position according to the Nth 3D depth image frame and a (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; The object tracking system of claim 3 further comprising:

5. The step of calculating the second 3D position comprises: calculating a target 3D depth image frame corresponding to a region of the object in the Nth 3D depth image frame; calculating the second 3D position according to the target 3D depth image frame and a (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; The object tracking system of claim 3 further comprising:

6. capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, the plurality of 2D image frames and the plurality of 3D depth image frames being interleaved, and a first 2D image frame of the plurality of 2D image frames being captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; calculating a second 3D position according to the N 3D depth image frame and an (N+1) 2D image frame of the plurality of 2D image frames; 3D position reporting method,

7. The 3D position reporting method of claim 6 , wherein the plurality of 2D image frames are near-infrared (NIR) image frames or RGB image frames.

8. The step of calculating the first 3D position comprises: calculating a plurality of 2D positions corresponding respectively to the plurality of 2D image frames; calculating the first 3D position according to an Nth 2D position corresponding to the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames; 7. The 3D position reporting method of claim 6, comprising:

9. The step of calculating the second 3D position comprises: calculating the second 3D position according to the Nth 3D depth image frame and a (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; 9. The 3D position reporting method of claim 8, further comprising:

10. The step of calculating the second 3D position comprises: calculating a target 3D depth image frame corresponding to a region of the object in the Nth 3D depth image frame; calculating the second 3D position according to the target 3D depth image frame and a (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames; 9. The 3D position reporting method of claim 8, further comprising:

Images