Object tracking system and 3D position reporting method

Abstract

A 3D position reporting method includes capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, where the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved with each other, the acquisition time of a first 2D image frame of the plurality of 2D image frames is earlier than that of a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N + 1) th 2D image frame of the plurality of 2D image frames.

Description

Technical Field

[0001] The present invention relates to an object tracking system and a 3D position reporting method, and in particular to an object tracking system and a 3D position reporting method for reporting a 3D position when each frame image is received. Background Art

[0002] As computer 3D vision applications mature and human-computer interaction sensing technology develops with the metaverse trend, smooth visual experience requires high frame rate tracking for applications such as virtual reality, augmented reality, and 3D naked vision. These applications require the use of RGB or grayscale images and depth images to track features such as faces, eyes, gestures, and postures to obtain the three-dimensional (3D) coordinates of landmarks on objects in the real world.

[0003] In this case, how to maximize the return rate of the landmark's 3D coordinates has become one of the industry's concerns. Summary of the invention

[0004] One of the main objectives of the present invention is to provide an object tracking system and a 3D position reporting method to solve the above-mentioned problems.

[0005] The present invention provides an object tracking system, comprising a sensor for capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; a processor coupled to the sensor; and a memory coupled to the processor for storing a program code to instruct the processor to execute a 3D position reporting method, wherein the 3D position reporting method comprises: calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames.

[0006] The present invention provides a 3D position reporting method, comprising: capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and calculating a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 FIG. 1 is a schematic diagram of an object tracking system according to an embodiment of the present invention.

[0008] Figure 2 FIG. 4 is a flow chart of a 3D position reporting method according to an embodiment of the present invention.

[0009] Figure 3 A timing diagram of a 3D position reporting method executed according to an embodiment of the present invention.

[0010] Figure 4A FIG. 1 is a schematic diagram of a 3D depth image frame DE of a human face tracked by the object tracking system according to an embodiment of the present invention.

[0011] Figure 4B FIG. 4 is a schematic diagram of a portion of a 3D depth image frame around a face according to an embodiment of the present invention.

[0012] The reference numerals are described as follows:

[0013] 1: Object tracking system

[0014] 10: Sensor

[0015] 20: Processor

[0016] 30: Memory

[0017] 2: Process

[0018] S200, S202, S204, S206: Steps DETAILED DESCRIPTION

[0019] Certain words are used in the specification and subsequent claims to refer to specific components. It should be understood by those with ordinary knowledge in the art that hardware manufacturers may use different terms to refer to the same component. This specification and subsequent claims do not use differences in names as a way to distinguish components, but use differences in the functions of the components as the criteria for distinction. The term "including" mentioned throughout the specification and subsequent claims is an open term and should be interpreted as "including but not limited to". In addition, the term "coupled" here includes any direct and indirect electrical connection means. Therefore, if the text describes a first device coupled to a second device, it means that the first device can be directly electrically connected to the second device, or indirectly electrically connected to the second device through other devices or connection means.

[0020] Please refer to Figure 1 , Figure 1 It is a schematic diagram of an object tracking system 1 according to an embodiment of the present invention. The object tracking system 1 includes a sensor 10, a processor 20 and a memory 30. The sensor 10 and the memory 30 are coupled to the processor 20, and are used to represent the basic elements of the object tracking system 1. The sensor 10 captures a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having a sensor frame rate. It should be noted that the plurality of 2D image frames may be near infrared (NIR) image frames, RGB image frames or grayscale image frames, etc., and are not limited thereto. The memory 30 stores a program code to instruct the processor 20 to execute a 3D position reporting method, so that the processor 20 generates a 3D position of a landmark of an object and maximizes a reporting rate of the 3D position.

[0021] The 3D position reporting method of the object tracking system 1 can be summarized into a process 2, such as Figure 2 Process 2 includes the following steps:

[0022] Step S200: Start.

[0023] Step S202: Calculate a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames.

[0024] Step S204: Calculate a second 3D position according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames.

[0025] Step S206: End.

[0026] Please refer to Figure 3 , Figure 3A timing diagram for executing a 3D position reporting method according to an embodiment of the present invention. In one embodiment, a plurality of 2D image frames and a plurality of 3D depth image frames are interleaved, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames. Specifically, the sensor 10 may capture a 2D image frame and a 3D depth image frame in each frame period corresponding to the sensor frame rate. For example, the sensor 10 captures the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames in the Nth frame period.

[0027] According to process 2, in step S202, the processor 20 calculates a first 3D position according to the Nth 2D image frame and the Nth 3D depth image frame. Figure 3 As shown, the processor 20 calculates the Nth 2D position corresponding to the Nth 2D image frame, and calculates the first 3D position based on the Nth 2D position and the Nth 3D depth image frame. In step S204, the processor 20 calculates the second 3D position based on the Nth 3D depth image frame corresponding to the Nth 2D image frame and the (N+1)th 2D position corresponding to the (N+1)th 2D image frame. In detail, the processor 20 stores the Nth 3D depth image frame calculated in the Nth frame period in the memory 30, and reuses the Nth 3D depth image frame calculated in the Nth frame period to calculate the second 3D position in the (N+1)th frame period. Similarly, the processor 20 can calculate a first 3D position and a second 3D position in each frame period corresponding to the sensor frame rate. In other words, the processor 20 can generate two 3D positions (a first 3D position and a second 3D position) of the landmark of the object in each frame period. In this way, the processor 20 can report a 3D position every half frame period, that is, the reporting rate of the processor 20 for the 3D position is twice the frame rate of the sensor.

[0028] It should be noted that the processor 20 can reuse the 3D depth image frame stored in any previous frame period to calculate the second 3D position in the current frame period, and is not limited thereto.

[0029] As described above, the 3D depth image frame needs to be retained until the next frame period and used together with the 2D position, but retaining the complete 3D depth image frame will result in high usage and long access time of the memory 30. In the present invention, the sensor 10 with a high sensor frame rate is used to ensure that the movement range of each image frame is limited to a specific range. Therefore, the present invention can optimize and retain a portion of the 3D depth image frame to calculate the 3D position of the landmark in the next frame period. Please refer to Figure 4A and Figure 4B, Figure 4A FIG. 1 is a schematic diagram of a 3D depth image frame of a human face tracked by the object tracking system 1 according to an embodiment of the present invention.

[0030] Figure 4B FIG. 1 is a schematic diagram of a portion of a 3D depth image frame around a face according to an embodiment of the present invention. It should be noted that the object tracking system 1 can track eyes on a face, but is not limited thereto. Figure 4B As shown, the processor 20 can calculate a target 3D depth image frame pair corresponding to an area of ​​a human face in a 3D depth image frame. For example, the area is a rectangular area surrounding the human eye, and the length and width of the rectangular area are h and w, respectively. Specifically, the processor 20 can calculate a second 3D position based on the target 3D depth image frame and the (N+1)th 2D position of the (N+1)th 2D image frame corresponding to multiple 2D image frames. In this way, the 3D position reporting method can reduce memory usage and access time by avoiding retaining the complete 3D depth image frame in the memory 30.

[0031] Furthermore, the processor 20 may select / determine at least one landmark, and obtain the parameter of the area corresponding to the at least one landmark according to the following equation:

[0032] X={x1,x2…x n}

[0033] Y={y1,y2…y n}

[0034] u=max(min(X)-ε w ,0)

[0035] v = max(min(Y) - ε h ,0)

[0036] w=min(max(X)-u+ε w ,Wu)

[0037] h=min(max(Y)-v+ε h ,Hv)

[0038] Wherein, (X, Y) represents the coordinates of at least one landmark; (u, v) represents the coordinates of a starting point of the region corresponding to at least one landmark; (w, h) represents the width and length of the region; and ε w and ε h They respectively represent the upper limits of the object's moving distance in the horizontal and vertical directions between two consecutive frame periods.

[0039] For example, if Figure 4A and Figure 4BAs shown, the processor 20 determines the human eye as two landmarks (x1, y1) and (x2, y2). According to the equation, the processor 20 obtains the area corresponding to the landmarks (x1, y1) and (x2, y2). In this way, the processor 20 can calculate the second 3D position according to the target 3D depth image frame corresponding to the area and the (N+1)th 2D position. In this way, the 3D position reporting method can reduce the memory usage and access time by avoiding retaining the complete 3D depth image frame in the memory 30.

[0040] It should be noted that the processor 20 can obtain multiple areas corresponding to multiple landmarks, rather than being limited to one area. For example, the processor 20 can obtain two areas corresponding to human eyes respectively.

[0041] A person with ordinary knowledge in the art may combine, modify or change the above-described embodiments according to the spirit of the present invention, but is not limited thereto. All of the above descriptions, steps, and / or processes (including recommended steps) can be implemented by hardware, software, firmware (i.e., a combination of hardware devices and computer instructions, the data in the hardware devices are read-only software data), electronic systems, or a combination of the above devices. Hardware may include analog, digital and hybrid circuits (i.e., microcircuits, microchips or silicon chips). Electronic systems may include system on chip (SoC), system in package (SiP), computer on module (CoM) and object tracking system 1. The process steps and embodiments of the present invention may exist in the form of program code or instructions and be stored in the memory 30. The memory 30 may be a computer-readable recording medium, and the memory 30 may include a read-only memory (ROM), a flash memory (Flash Memory), a random-access memory (RAM), a subscriber identity module (SIM), a hard disk or a CD-ROM / DVD-ROM / BD-ROM, but is not limited thereto. The above-mentioned processes and embodiments may be compiled into program codes or instructions and stored in the memory 30. The processor 20 may be used to read and execute the program codes or instructions stored in the memory 30 to implement all the aforementioned steps and functions.

[0042] In summary, the prior art only reports one 3D position during each frame period. In contrast, the 3D position reporting method of the present invention can report two 3D positions during each frame period. In other words, the reporting rate of the present invention is twice that of the prior art. In this way, the present invention maximizes the reporting rate of the 3D position and achieves a smooth visual experience.

[0043] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the claims of the present invention should fall within the scope of the present invention.

Claims

1. An object tracking system, comprising: a sensor for capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interlaced with each other, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; a processor coupled to the sensor; and A memory, coupled to the processor, for storing a program code to instruct the processor to execute a 3D position reporting method, wherein the 3D position reporting method includes: Calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and A second 3D position is calculated according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames. 2 . The object tracking system as claimed in claim 1 , wherein the plurality of 2D image frames are near infrared image frames or RGB image frames.

3. The object tracking system of claim 1 , wherein the step of calculating the first 3D position comprises: Calculating a plurality of 2D positions corresponding to the plurality of 2D image frames respectively; and The first 3D position is calculated according to an Nth 2D position corresponding to the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames.

4. The object tracking system as claimed in claim 3, wherein the step of calculating the second 3D position further comprises: The second 3D position is calculated according to the Nth 3D depth image frame and an (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames.

5. The object tracking system as claimed in claim 3, wherein the step of calculating the second 3D position further comprises: calculating a target 3D depth image frame corresponding to an area of ​​an object located in the Nth 3D depth image frame; and The second 3D position is calculated according to the target 3D depth image frame and an (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames.

6. A 3D position reporting method, comprising: capturing a plurality of 2D image frames having a sensor frame rate and a plurality of 3D depth image frames having the sensor frame rate, wherein the plurality of 2D image frames and the plurality of 3D depth image frames are interlaced with each other, and a first 2D image frame of the plurality of 2D image frames is captured earlier than a first 3D depth image frame of the plurality of 3D depth image frames; Calculating a first 3D position according to an Nth 2D image frame of the plurality of 2D image frames and an Nth 3D depth image frame of the plurality of 3D depth image frames; and A second 3D position is calculated according to the Nth 3D depth image frame and an (N+1)th 2D image frame of the plurality of 2D image frames. 7 . The 3D position reporting method as claimed in claim 6 , wherein the plurality of 2D image frames are near infrared image frames or RGB image frames.

8. The 3D position reporting method as claimed in claim 6, wherein the step of calculating the first 3D position comprises: Calculating a plurality of 2D positions corresponding to the plurality of 2D image frames respectively; and The first 3D position is calculated according to an Nth 2D position corresponding to the Nth 2D image frame of the plurality of 2D image frames and the Nth 3D depth image frame of the plurality of 3D depth image frames.

9. The 3D position reporting method as claimed in claim 8, wherein the step of calculating the second 3D position further comprises: The second 3D position is calculated according to the Nth 3D depth image frame and an (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames.

10. The 3D position reporting method as claimed in claim 8, wherein the step of calculating the second 3D position further comprises: calculating a target 3D depth image frame corresponding to an area of ​​an object located in the Nth 3D depth image frame; and The second 3D position is calculated according to the target 3D depth image frame and an (N+1)th 2D position corresponding to the (N+1)th 2D image frame of the plurality of 2D image frames.

Images