Object tracking method and processing device and system

Abstract

An object tracking method and a processing device and system thereof, the method comprising the following steps: using a main camera and at least one auxiliary camera to acquire images of a physical scene to generate a main image and at least one auxiliary image respectively; performing image superposition on the main image and the auxiliary image to generate a superimposed image; calculating the position information and depth information of each object in the superimposed image; and judging whether all objects in the superimposed image have the same object according to the position information and depth information of each object.

Description

Technical Field

[0001] The present invention relates to a technology for object tracking. Background Art

[0002] With the development of image processing technology and spatial positioning technology, the application of interactive displays has gradually received attention. This type of technology can produce an interactive experience according to the user's operational needs, so that information can be presented in a more intuitive way. However, the field of view covered by a single-camera system is limited and cannot meet the needs of a wide range of operations. The fisheye camera system will deform the image, making it difficult to recognize the face. Although the multi-camera system can cover a larger field of view, in cross-field face tracking, in addition to the need to perform a large amount of feature comparison, image stitching, and image recognition operations on the overlapping areas of the field of view, there will also be concerns about repeated calculations of faces. Summary of the invention

[0003] The present invention provides an object tracking method and a processing device and system thereof, which can provide object tracking with high recognition rate and efficiency in the field of view of different cameras and can effectively avoid repeated calculation of the same object.

[0004] In an exemplary embodiment of the present invention, the method includes the following steps: using a main camera and at least one auxiliary camera to acquire images of a physical scene to generate a main image and at least one auxiliary image respectively; performing image superposition on the main image and the auxiliary image to generate a superimposed image; calculating the position information and depth information of each object in the superimposed image; and judging whether all objects in the superimposed image have the same object according to the position information and depth information of each object.

[0005] In an exemplary embodiment of the present invention, the processing device is connected to a main camera and at least one auxiliary camera, and includes a memory and a processor. The memory is used to store data. The processor is used to receive a main image and at least one auxiliary image respectively generated by image acquisition for a physical scene from the main camera and at least one auxiliary camera, and then perform image superposition on the main image and at least one auxiliary image to generate a superimposed image, and calculate the position information and depth information of each object in the superimposed image, and determine whether there is an identical object among all objects in the superimposed image according to the position information and depth information of each object.

[0006] In an exemplary embodiment of the present invention, the system includes a main camera, at least one auxiliary camera, and a processing device. The main camera is used to acquire images of a physical scene to generate a main image. The at least one auxiliary camera is used to acquire images of a physical scene to generate at least one auxiliary image. The processing device is connected to the main camera and the at least one auxiliary camera, and is used to obtain a main image and at least one auxiliary image, and then perform image superposition on the main image and the at least one auxiliary image to generate a superimposed image, and calculate the position information and depth information of each object in the superimposed image, and determine whether all objects in the superimposed image have the same object according to the position information and depth information of each object.

[0007] In order to make the present invention more clearly understood, embodiments are given below with reference to the accompanying drawings for detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 FIG. 4 is a schematic diagram of an object tracking system according to an exemplary embodiment of the present invention.

[0009] Figure 2 is a flow chart of an object tracking method according to an exemplary embodiment of the present invention.

[0010] Figure 3 is a schematic diagram of an application scenario of an object tracking method according to an exemplary embodiment of the present invention.

[0011] Figure 4A as well as Figure 4B FIG. 4 is a schematic diagram of a method for calculating pixel displacement according to an exemplary embodiment of the present invention.

[0012] Figure 5A is a schematic diagram of a camera installation method according to an exemplary embodiment of the present invention.

[0013] Figure 5B FIG. 4 is a schematic diagram of a method for calculating pixel displacement according to an exemplary embodiment of the present invention.

[0014] Figure 6 is a schematic diagram of a multi-camera system installation method according to an exemplary embodiment of the present invention.

[0015] Fig. 7A FIG. 4 is a flowchart of a method for selecting a main camera according to an exemplary embodiment of the present invention.

[0016] Figure 7B is a schematic diagram of an application scenario of selecting a main camera according to an exemplary embodiment of the present invention.

[0017] Figures 8A to 8D FIG. 4 is a schematic diagram of a method for shortening a depth detection limit according to an exemplary embodiment of the present invention.

[0018] Fig. 9 is a flow chart of an object tracking method according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0019] Some exemplary embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. The referenced element symbols in the following description will be regarded as the same or similar elements when the same element symbols appear in different drawings. These exemplary embodiments are only part of the present invention and do not disclose all possible implementation methods of the present invention. More specifically, these exemplary embodiments are only examples of the methods, devices, and systems in the claims of the present invention.

[0020] Figure 1 FIG. 1 is a schematic diagram of an object tracking system according to an exemplary embodiment of the present invention. Figure 1 First, we will introduce the components and configuration relationships in the system. The detailed functions will be matched with the process of the subsequent example implementation. Figure 1 And reveal.

[0021] Please refer to Figure 1 The object tracking system 100 includes a main camera 110, at least one auxiliary camera 120, and a processing device 130, wherein the processing device 130 may be wirelessly, wired, or electrically connected to the main camera 110 and the auxiliary camera 120. It should be noted that in this exemplary embodiment, only a single auxiliary camera 120 is illustrated as an example, and in other exemplary embodiments, the auxiliary camera 120 may be plural.

[0022] The main camera 110 and the auxiliary camera 120 may include an image sensor and a depth sensor, that is, the main camera 110 and the auxiliary camera 120 may have the functions of image acquisition and depth acquisition. The image sensor is used to acquire images, and it is, for example, a camera lens having a lens and a photosensitive element. The photosensitive element is used to sense the intensity of light entering the lens, and then generate an image. The photosensitive element may be, for example, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) element or other similar elements. The depth sensor is used to detect depth information, which can be divided into active depth sensing technology and passive depth sensing technology. Active depth sensing technology can calculate depth information by actively emitting light sources, infrared rays, ultrasonic waves, lasers, etc. as signals in combination with time difference ranging technology. Passive depth sensing technology can use two image sensors to simultaneously acquire two images in front of it at different viewing angles, so as to calculate depth information using the parallax of the two images.

[0023] The processing device 130 is used to process the images acquired by the main camera 110 and the auxiliary camera 120 to execute the processes in multiple exemplary embodiments of the present invention. The processing device 130 includes a memory 132 and a processor 134. The memory 132 can be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar devices, integrated circuits or combinations thereof. The processor 134 can be, for example, a central processing unit (CPU), an application processor (AP) or other programmable general-purpose or special-purpose microprocessor (microprocessor), a digital signal processor (DSP), an image signal processor (ISP), a graphics processor (GPU) or other similar devices, integrated circuits or combinations thereof.

[0024] Figure 2 is a flow chart of an object tracking method according to an exemplary embodiment of the present invention, and Figure 2 The method flow can be obtained by Figure 1 The object tracking system 100 is implemented.

[0025] Please also refer to Figure 1 as well as Figure 2 First, the main camera 110 and at least one auxiliary camera 120 will acquire images of the physical scene to generate a main image and at least one auxiliary image respectively (step S202). That is, the main camera 110 and the auxiliary camera 120 will synchronously acquire images of the physical scene from different perspectives to generate images with different field of view ranges. Next, the processing device 130 will perform image superposition on the main image and the auxiliary image to generate a superimposed image (step S204), and will calculate the position information and depth information of each object in the superimposed image (step S206). The image superposition here can be horizontal superposition (left-right superposition) or vertical superposition (up-down superposition) of the main image and the auxiliary image.

[0026] In an exemplary embodiment, the processing device 130 can determine whether the main camera 110 and the auxiliary camera 120 are horizontally photographed or vertically photographed to determine the image superposition method, so as to reduce the influence of the deformation caused by the image superposition on the recognition rate. For example, when the main camera 110 and the auxiliary camera 120 are horizontally photographed and the camera resolution is 640×360, the image resolution after horizontal superposition is 1280×360, and the image resolution after vertical superposition is 640×720. Generally speaking, before performing face recognition, the processing device 130 scales the superimposed image to an image resolution of 300×300, for example, the horizontal superimposed superimposed image will produce serious deformation, resulting in a decrease in the recognition rate. Take two videos with a resolution of 640×360 and a length of 74 seconds as an example, and perform face recognition after superimposing them separately. From the recognition results, it can be seen that the recognition rate of vertical superposition is higher. Therefore, when the main camera 110 and the auxiliary camera 120 are in horizontal shooting mode, the processing device 130 will vertically overlap the main image and the auxiliary image to generate a superimposed image. When the main camera 110 and the auxiliary camera 120 are in vertical shooting mode, the processing device 130 will horizontally overlap the main image and the auxiliary image to generate a superimposed image.

[0027] Next, the processing device 130 will calculate the position information and depth information of each object in the superimposed image (step S206), and will determine whether there is the same object among all the objects in the superimposed image based on the position information and depth information of each object (step S208). For example, when the processing device 130 determines that two objects have a corresponding relationship in the position and depth corresponding to the main image and the auxiliary image in the superimposed image, the two objects will be regarded as the same object. In other words, this example embodiment uses image superposition instead of image stitching to perform cross-field object tracking, which can achieve the effect with less computing power and computing time. Afterwards, the processing device 130 can perform subsequent processing on the same object depending on the actual application scenario.

[0028] For the sake of convenience and clarity, the following exemplary embodiments will be specifically described with reference to different application scenarios of the object tracking system 100. Figure 2 Details of the process.

[0029] Figure 3 This is an application scenario of an object tracking method according to an exemplary embodiment of the present invention. In this exemplary embodiment, it is assumed that the main camera 110 and the auxiliary camera 120 of the object tracking system 100 are two cameras set up in parallel, and the tracking object is a human face.

[0030] Please refer to Figure 1 as well as Figure 3, the main camera 110 and the auxiliary camera 120 will acquire images of the physical scene to generate a main image 300A and an auxiliary image 300B, respectively (step S302). Next, the processing device 130 will perform image superposition on the main image 300A and the auxiliary image 300B to generate a superimposed image 310 (step S304). In this exemplary embodiment, the processing device 130 performs image superposition on the main image 300A and the auxiliary image 300B in a vertical superposition manner, wherein the main area 310A and the auxiliary area 310B in the superimposed image 310 correspond to the main image 300A and the auxiliary image 300B, respectively.

[0031] Next, the processing device 130 will identify each region of the face from the superimposed image, and calculate the position information and depth information of each face, so as to determine whether the main region 310A and the auxiliary region 310B in the superimposed image 310 have the same face. Here, the processing device 130 will determine whether the main region 310A and the auxiliary region 310B in the superimposed image 310 have the same face based on three conditions: vertical coordinate, depth information, and horizontal coordinate (step S306).

[0032] In this example embodiment, the processing device 130 may first make a vertical coordinate judgment. When the processing device 130 determines that the vertical coordinate of the face 31A in the main area 310A is the same as the vertical coordinate of the face 31B in the auxiliary area 310B, a depth judgment will be made. When the processing device 130 determines that the depth of the face 31A is the same as the depth of the face 31B, a horizontal coordinate judgment will be made. Next, the processing device 130 will determine whether the difference between the horizontal coordinate of the face 31A in the main area 310A and the horizontal coordinate of the face 31B in the auxiliary area 310B is the same as the pixel displacement ΔX. Here, the processing device 130 will calculate the pixel displacement ΔX based on the horizontal resolution of the auxiliary camera 120, the horizontal field of view distance, and the lens spacing between the main camera 110 and the auxiliary camera 120. The following Figure 4A as well as Figure 4B A schematic diagram of a method of calculating the pixel shift amount ΔX will be specifically described.

[0033] Please refer to Figure 4A , considering the position of the depth Z (in mm) relative to the camera 420, according to the definition of the horizontal field of view FOV of the camera 420, the horizontal field of view distance FOVX (in mm) can be calculated as shown in equation (1):

[0034]

[0035] Please refer to Figure 4B, when a camera moves horizontally by a specific distance, for a stationary object, it is equivalent to the camera being stationary and the object moving the same distance in the opposite direction. From another point of view, when two cameras are separated by a specific distance, calculating the difference ΔX (the correction value required for horizontal pixels) in the horizontal pixel coordinates of the same object 42 in the field of view F2 and F1 captured by the two cameras is equivalent to calculating the pixel distance between two objects separated by a specific distance captured by the same camera. Therefore, the above concept can be obtained by the following two equations (2) and (3):

[0036]

[0037] Where RSX is the horizontal resolution, and L is the distance between the two lenses. If equations (2) and (3) are combined and then substituted into equation (3), equation (4) for calculating the pixel displacement ΔX can be obtained:

[0038]

[0039] That is:

[0040]

[0041] For example, assuming that the distance L between the main camera and the auxiliary camera is 15 cm, the image resolution RSX is 640×360, and the horizontal field of view FOV of the camera is 75 degrees, the pixel displacement ΔX of the auxiliary camera's pixel conversion to the corresponding pixel of the main camera is as shown in equation (5):

[0042]

[0043] In order to reduce the amount of calculation, the above equation (5) can be expanded polynomially around the depth Z = 600mm according to the Taylor series of equation (5.1) (i.e., a = 600 is substituted), and only the first three terms are considered to generate equation (5.2). The reason for using the depth Z = 600mm here is that the error at the near point (Z = 200mm) and the far point (Z = 1000mm) will be too large:

[0044]

[0045] in:

[0046]

[0047] Therefore, equation (5.2) can be written as the following equation (5.3):

[0048]

[0049] If equation (5.3) is expanded, we can obtain the quadratic polynomial of equation (6):

[0050] f(z)=ΔX=0.00039Z 2 -0.7Z+417 (6)

[0051] Please come back Figure 3 , after the processing device 130 calculates the pixel displacement ΔX, it will determine the horizontal coordinates. When the processing device 130 determines that the difference between the horizontal coordinates of face 31A and the horizontal coordinates of face 31B is the same as the pixel displacement ΔX, it means that face 31A and face 31B are the same face. Taking this exemplary embodiment as an example, after the processing device 130 completes the judgment of step S306 for other faces, it will determine that faces 31A and 32A in the main area 310A are the same faces as faces 31B and 32B in the auxiliary area 310B, respectively. Therefore, the processing device 130 can perform subsequent processing on faces 31B and 32B acquired by the auxiliary camera 120B, such as determining that faces 31B and 32B are repeated without performing subsequent related calculations (step S308).

[0052] It should be noted that those skilled in the art should understand that the aforementioned processing device 130 determines whether the difference between the horizontal coordinates of the face 31A and the horizontal coordinates of the face 31B is the same as the pixel displacement ΔX under the premise of whether there is an allowable tolerance threshold. For example, when the image resolution is 640×360 and the lens distance L between the main camera and the auxiliary camera is 10 cm, the pixel coordinates (px, py) of the auxiliary camera are converted to the pixel coordinates (px', py') of the main camera using equation (6.5):

[0053]

[0054] Where ΔX = 5.83 × 10 -4 Z 2 -1.052Z+570.7, and the unit of Z is mm. Assuming that the distance between the pixel coordinates (px', py') of the face 31B in the auxiliary area 310B after being converted to the main area 310A and the pixel coordinates of the face 31A in the main area 310A is less than the distance threshold d, the face 31B will be determined to be a repeated face. Assuming that there is no face within the range of distance d from the pixel coordinates (px', py') in the main area 310A, the face 31B will be determined to be a non-duplicate face.

[0055] In another example embodiment, the processing device 130 may first determine the depth. When the processing device 130 determines that the depth of the face 31A in the main area 310A is the same as the depth of the face 31B in the auxiliary area 310B, the vertical coordinate determination will be performed. When the processing device 130 determines that the vertical coordinate of the face 31A is the same as the vertical coordinate of the face 31B, the horizontal coordinate determination will be performed. The details of this example embodiment can be referred to the above related description and will be inferred by analogy, which will not be repeated here.

[0056] It should be noted that those skilled in the art should understand that the aforementioned determination of whether the three conditions of vertical coordinates, depth information, and horizontal coordinates of face 31A and face B (hereinafter referred to as "vertical conditions", "depth conditions", and "horizontal conditions") are met is based on the premise of an allowable tolerance threshold. Specifically, determining whether the vertical coordinates of face 31A in the main area 310A and face 32B in the auxiliary area 310B are the same refers to determining whether the difference between the vertical coordinates of face 32B in the auxiliary area 310B and the vertical coordinates of face 31A in the main area 310A (hereinafter referred to as "vertical difference") is less than the vertical tolerance threshold. If so, it is determined that the vertical condition is met. Determining whether the depths of face 31A in the main area 310A and face 32B in the auxiliary area 310B are the same refers to determining whether the difference between the depth of face 32B in the auxiliary area 310B and the depth of face 31A in the main area 310A (hereinafter referred to as "depth difference") is less than the depth tolerance threshold. If so, it is determined that the depth condition is met. Determining whether the difference between the horizontal coordinates of the face 31A in the main area 310A and the face 32B in the auxiliary area 310B and the pixel displacement is the same means determining whether the difference between the horizontal coordinates of the face 32B in the auxiliary area 310B and the horizontal coordinates of the face 31A in the main area 310A (hereinafter referred to as "horizontal difference") and the pixel displacement is less than a horizontal tolerance threshold. If so, it is determined that the horizontal condition is met.

[0057] The above tolerance threshold may depend on the actual size of the face or the distance of the eyes in the image. Specifically, considering that two people may be close together, the depth tolerance threshold may be smaller than the diameter of the head (e.g., 16 cm). Considering the need for a dominant eye, the horizontal tolerance threshold may be smaller than the pixel distance between the left and right eyes at the limit distance, while the vertical coordinate may be comparable to the horizontal coordinate. For example, the distance between the left and right eyes (pupillary distance) is 7-8 cm, when the camera resolution is 640×360, the camera horizontal field of view is 75 degrees, and the limit distance is 90 cm, the pixel distance between the left and right eyes is 80 pixels. Further, considering the side face or face overlap, the optimal tolerance threshold may be 1 / 4 of the general case. For example: the optimal depth tolerance threshold may be set to 40 mm, the optimal horizontal tolerance threshold and the optimal vertical tolerance threshold may be set to 20 pixels.

[0058] Figure 3 as well as Figure 4A-4B This is the case when the main camera and the auxiliary camera are mounted horizontally and their optical axes are parallel. When the main camera and the auxiliary camera are mounted horizontally but the optical axes have an inclination angle or an outclination angle, the calculation method of the pixel displacement will be different. Figure 5A According to a schematic diagram of a camera installation method depicted in an exemplary embodiment of the present invention, the main camera 510 and the auxiliary camera 520 both have an inclination angle θ, and the distance between the two lenses is L. The calculation method of the pixel displacement can be divided into Figure 5B The three steps are illustrated according to an exemplary embodiment of the present invention.

[0059] Please refer to Figure 5B From the perspective of the auxiliary camera 520, the calculation of the pixel displacement can be divided into the translation of the auxiliary camera 520 (step S502), the auxiliary camera 520 rotates 2θ in situ (step S504), and the auxiliary camera 520 continues to translate (Step S506). Here, when the rotation angle of the auxiliary camera 520 is θ, the horizontal pixel displacement ΔX can be expressed by equation (7):

[0060]

[0061] Then, with equation (4), the camera translation calculation formula can be used to estimate the pixel displacement of the auxiliary camera 520 in this embodiment. The corresponding horizontal pixel displacement ΔX1 can be expressed by equation (8.1):

[0062]

[0063] The horizontal pixel displacement ΔX2 corresponding to the in-situ rotation of the auxiliary camera 520 by 2θ can be expressed by equation (8.2):

[0064]

[0065] Translation of auxiliary camera 520 The corresponding horizontal pixel displacement ΔX3 can be expressed by equation (8.3):

[0066]

[0067] Therefore, the total pixel displacement of the auxiliary camera 520 can be expressed by equation (8.4):

[0068]

[0069] Similarly, when both the main camera and the auxiliary camera have an outward tilt angle θ′, from the perspective of the auxiliary camera 520, the calculation of the pixel displacement can also be divided into the translation of the auxiliary camera 520 The auxiliary camera 520 rotates 2θ' in situ and the auxiliary camera 520 continues to translate In detail, the translation of the auxiliary camera 520 The corresponding horizontal pixel displacement ΔX1 can be expressed by equation (9.1):

[0070]

[0071] The horizontal pixel displacement ΔX2 corresponding to the in-situ rotation 2θ′ of the auxiliary camera 520 can be expressed by equation (9.2):

[0072]

[0073] Translation of auxiliary camera 520 The corresponding horizontal pixel displacement ΔX3 can be expressed by equation (9.3):

[0074]

[0075] Therefore, the total pixel displacement of the auxiliary camera 520 can be expressed by equation (9.4):

[0076]

[0077] When two cameras are not enough to cover the user's interaction area, cameras can be added to the system to form a modular camera array. In a modular camera array, there will be more than three cameras. Figure 6 is a schematic diagram of a multi-camera system installation method according to an exemplary embodiment of the present invention, and the basis for selecting a main camera will be described below.

[0078] Please refer to Figure 6 , assuming that camera 610A, camera 610B and camera 610C are three cameras arranged horizontally, and RAB is the overlapping area of ​​camera 610A and camera 610B, RBC is the overlapping area of ​​camera 610B and camera 610C, and RAC is the overlapping area of ​​camera 610A and camera 610C. Here, the sum of the overlapping area of ​​each camera and the closest distance S1 and S2 of the camera will be calculated, and the camera with the smallest sum is set as the main camera, and the remaining cameras are set as auxiliary cameras. The distances between each overlapping area and the camera are shown in Table 1:

[0079] Camera 610A Camera 610B Camera 610C Camera 610A - S2 S1 Camera 610B S2 - S2 Camera 610C S1 S2 - sum S1+S2 2S2 S1+S2

[0080] Table 1

[0081] The sum of distances of the overlapping areas of camera 610B is the smallest, so camera 610B can be set as the main camera, and camera 610A and camera 610C can be regarded as auxiliary cameras.

[0082] In addition, the closer the face is to the camera, the greater the pixel displacement between the cameras, so the error in coordinate conversion will also be greater. Therefore, the fewer coordinate conversions at close distances can also be another basis for selecting the main camera. Figure 6 Taking camera A and camera B as an example, the following Table 2 lists the number of coordinate transformations required for the position P1 of the face in the overlapping area RAB and the position P2 of the overlapping area RBC:

[0083]

[0084] Table 2

[0085] The total number of coordinate transformations of camera B is the smallest, so camera B can be set as the main camera, and cameras A and C can be regarded as auxiliary cameras.

[0086] When used at short distances, if one camera cannot obtain depth information, it will be necessary to dynamically select the main camera. Fig. 7A is a flowchart of a method for selecting a main camera according to an exemplary embodiment of the present invention. Figure 7B is a schematic diagram of an application scenario of selecting a main camera according to an exemplary embodiment of the present invention.

[0087] Please refer to Figure 1 as well as Fig. 7A , camera 710A and camera 710B will respectively acquire images (step S702A, step S702B), and the processing device 130 will then perform face recognition on the acquired images (step S704A, step S704B), and further acquire the depth of the location of the face (step S706A, step S706B). When the depth acquisition of camera 710A fails, the processing device 130 will set camera 710A as the auxiliary camera and camera 710B as the main camera (step S710A). When the depth acquisition of camera 710B fails, the processing device 130 will set camera 710B as the auxiliary camera and camera 710A as the main camera (step S710B).

[0088] When the depth acquisition of camera 710A and camera 710B is successful, the processing device 130 can further determine the size of depth A and depth B, for example, whether depth A is greater than or equal to depth B (step S710). When depth A is less than depth B, it means that the position corresponding to this depth is closer to camera A, and the processing device 130 will set camera 710B as the main camera and camera 710A as the auxiliary camera (step S710A). When depth A is greater than or equal to depth B, it means that the position corresponding to this depth is closer to camera 710B or is located at the center of camera 710A and camera 710B, and the processing device 130 will set camera 710B as the auxiliary camera and camera 710A as the main camera (step S710B).

[0089] Please refer to Figure 7B , user 71 is located in the center of the two cameras, then camera A will be set as the primary camera for user 71. User 72 is closer to camera 710A, then camera 710B will be set as the primary camera for user 72. User 73 is closer to camera 710B, then camera 710A will be set as the primary camera for user 73.

[0090] Each camera has a depth detection limit. Assuming the depth detection limit is 20cm, it means that the camera cannot measure the depth of objects within 20cm. However, by setting up multiple cameras, the depth detection limit can be reduced and objects closer can be measured. Figures 8A to 8D A schematic diagram of a method for shortening the depth detection limit is illustrated according to an exemplary embodiment of the present invention.

[0091] Please refer to Fig. 8A For example, the diameter of the user's head H is 16 cm. When the distance between the center of the eyebrows H1 and the camera 810A is 20 cm, the distance between the center O of the head H and the camera 810A will be 28 cm. At this time, the width of the face will occupy 32 degrees of the horizontal field of view.

[0092] Please refer to Figure 8B , the horizontal field of view of cameras 810A and 810B is 75°, the depth detection limit is 20cm, the camera distance is 22.5cm and they are all tilted inward at 13°. Assume that the user stands in the middle of cameras 810A and 810B, and the distance between the center of the eyebrows H1 and the baseline BL is 10cm. Assume that the center of the head H is point O, the center point of the two cameras is point B, and the center of the lens of camera 810 is point C. There will be point A on the vertical baseline through point C. The direction of the CD segment is the direction of the optical axis of camera 810A, and point D is the vertical projection point of point O on the CD segment.

[0093] Here, the length of line segment BC is half of the camera distance, which is 11.25 cm. The length of line segment OB, the distance from the center point O of the human head to the baseline BL, is 18 cm. Since the camera 810A is tilted inwards by 13°, the angle ∠OCD is 13°. Through trigonometric functions, it can be deduced that the angle ∠OCB is 58°, the length of the hypotenuse OC is 21.23 cm, and the angle ∠OCD is 19°. Finally, it can be calculated that the length of line segment CD is 28 cm, and the depth of the human head surface is 20 cm. As Figure 8C , the user's head can be completely within the field of view of the camera 810A and is detectable. Therefore, by setting up the cameras in this way, it is possible to reduce the depth detection limit of the multi-camera system to 10 cm, making the depth detection limit better than that of a single-camera system.

[0094] Briefly speaking, referring to Figure 8C and Fig.8D , when the horizontal field of view of a single camera is fov and the depth detection limit is d1, the horizontal setup and the inward tilt angle θ enable the minimum depth detection limit of the multi-camera system to be reduced to d2, where d2 < d1, and the preset distance L between the two cameras will satisfy the following equation (9):

[0095]

[0096] Incidentally, when the user is near the center point between the camera 810A and the camera 810B, the spatial coordinates of the eyeballs can be obtained through the camera 810A and the camera 810B. After separately converting the coordinates to the new point (point B) on the baseline BL, and then through the face model, the spatial coordinates of the center of the eyebrows H1 can be estimated, and thus virtual-real interaction with the transparent screen can be achieved. Taking Fig. 8A and Figure 8B as an example, at an image resolution of 640×360, the oblique angle θ can be estimated based on the horizontal coordinate x as . The distance from the camera can be approximated as the depth value of the human face in the camera. Then, adding the 13-degree camera tilt angle and the oblique angle, the angle between the human face and the baseline is obtained, that is, the angle is (90° - inward tilt angle - θ). Then, multiplying the depth by the sine value of the angle gives the distance from the human face to the baseline, which is the converted depth value. Additionally, multiplying the depth by the cosine value of the angle gives the distance from the projection point of the human face to the camera, which is the converted horizontal coordinate.

[0097] When the user is far from the center point between the camera 810A and the camera 810B, only one camera can obtain the depth information of the ipsilateral eye, and the other camera can only obtain the pixel position of the other eye. At this time, the nearby camera can be used to roughly estimate the user's distance based on the size and angle of the human face, and then perform virtual-real interaction with the user.

[0098] The above exemplary embodiments can utilize Fig. 9According to the flowchart of the object tracking method depicted in an exemplary embodiment of the present invention, Fig. 9 The method flow can be obtained by Figure 1 The object tracking system 100 is implemented.

[0099] Please refer to Fig. 9 First, the main camera 110 and the auxiliary camera 120 will acquire images according to the synchronization signal (step S902A, step S902B), and the processing device 130 will overlap the images acquired by the main camera 110 and the auxiliary camera 120 to generate a superimposed image (step S904). Next, the processing device 130 will perform face recognition on the main area and the auxiliary area in the superimposed image (step S906A, step S906B), and obtain depth data from the main camera 110 and the auxiliary camera 120 (steps S908A, S908B). For the main area corresponding to the main camera 110, the processing device 130 will perform a gaze point calculation (step S910A) and generate a calculation result (step S912A). For the auxiliary area corresponding to the auxiliary camera 120, the processing device 130 will perform a valid face determination (step S909B), that is, determine whether there is a face in the auxiliary area that is the same as that in the main area. Afterwards, for the auxiliary area corresponding to the auxiliary camera 120, the processing device 130 will perform gaze point calculation for the valid face (step S910B) and generate a calculation result (step S912B). Afterwards, the processing device 130 will perform subsequent processing on the calculation results of step S912A and step S912B (step S914), and this subsequent processing will be set according to the application scenario. For example, the processing device 130 will display subsequent virtual-reality interaction information according to the calculation result.

[0100] Table 3 below lists the speed differences between the object tracking system using image stitching and the object tracking system 100 using image overlap for cross-view object tracking:

[0101]

[0102] Table 3

[0103] It can be seen from Table 3 that the time for image superposition is much shorter than the time for image stitching, so the object tracking system 100 can provide a faster object tracking method.

[0104] The embodiment of the present invention provides an object tracking method and a processing device and system thereof, which can provide object tracking with high recognition rate and efficiency in different camera fields of view and can effectively avoid repeated calculation of the same object.

[0105] Although the present invention has been disclosed as above by way of example embodiments, it is not intended to limit the present invention. Any person skilled in the art may make slight changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the appended claims and their equivalents.

[0106] [Explanation of symbols]

[0107] 100: Object Tracking System

[0108] 110: Main camera

[0109] 120: Auxiliary camera

[0110] 130: Processing device

[0111] 132: Memory

[0112] 134: Processor

[0113] S202-S208, S302-S308, S702A-S710B, S902A-S914: Step 300A: Main Image

[0114] 300B: Auxiliary image

[0115] 310A: Main area

[0116] 310B: Auxiliary area

[0117] 310: Overlay images

[0118] 31A, 31B, 32A, 32B: Face

[0119] ΔX: pixel displacement

[0120] 420, 510, 520, 610A, 610B, 610C, 710A, 710B, 810A, 810B: Camera FOV, F1, F2: Field of view

[0121] FOVX: horizontal field of view distance

[0122] Z: Depth

[0123] 42: Object

[0124] L: Lens distance

[0125] θ: inclination angle

[0126] RAC, RAB, RBC: Overlapping area of ​​vision

[0127] S1, S2: Distance

[0128] H: Head

[0129] O: Head Center

[0130] H1: Between the eyebrows

[0131] A, B, C, D, O: Point

[0132] BL: Baseline

[0133] fov: horizontal field of view

[0134] d1, d2: Depth detection limits

Claims

1. An object tracking method, characterized in that: include: Using a main camera and at least one auxiliary camera to acquire images of a physical scene, so as to generate a main image and at least one auxiliary image respectively; Performing image superposition on the main image and the auxiliary image to generate a superimposed image; Calculating the position information and depth information of each object in the superimposed image; as well as According to the position information and the depth information of each of the objects, it is determined whether the objects in the superimposed image are the same object. The step of calculating the position information and the depth information of each object in the superimposed image includes: recognizing each of the objects from the superimposed image, wherein the superimposed image includes a primary area corresponding to the primary camera and a secondary area corresponding to the secondary camera; and Determine the vertical coordinates, horizontal coordinates and depth of each of the objects located in the main area and the auxiliary area, The object includes a first object located in the main area, and the step of determining whether the objects in the superimposed image are identical objects according to the position information and the depth information of each object includes: Determine whether the secondary region has a second object whose vertical coordinate, depth, and horizontal coordinate meet the vertical condition, depth condition, and horizontal condition respectively with the vertical coordinate, the depth, and the horizontal coordinate of the first object; When the vertical condition, the depth condition, and the horizontal condition are all met, determining that the objects in the superimposed image are identical objects; and When any one of the vertical condition, the depth condition, and the horizontal condition is not satisfied, it is determined that the objects in the superimposed image do not have the same object.

2. The method according to claim 1, characterized in that The step of performing image superposition on the main image and the auxiliary image to generate the superimposed image includes: Determine whether the main camera and the auxiliary camera are in vertical shooting mode or horizontal shooting mode; When the main camera and the auxiliary camera are in the vertical shooting mode, horizontally superimposing the main image and the auxiliary image to generate the superimposed image; and When the main camera and the auxiliary camera are in the horizontal shooting mode, the main image and the auxiliary image are vertically overlapped to generate the overlapped image.

3. The method according to claim 1, characterized in that When the vertical difference between the vertical coordinate of the second object and the vertical coordinate of the first object is less than a vertical tolerance threshold, it is determined that the vertical condition is met.

4. The method according to claim 3, characterized in that When the first object and the second object are faces, the vertical tolerance threshold is associated with a pixel distance between left and right eyes.

5. The method according to claim 1, characterized in that When the depth difference between the depth of the second object and the depth of the first object is less than a depth tolerance threshold, it is determined that the depth condition is met.

6. The method according to claim 5, characterized in that When the first object and the second object are human faces, the depth tolerance threshold is associated with a diameter of a human head.

7. The method according to claim 1, characterized in that When the horizontal difference between the horizontal coordinate of the second object and the horizontal coordinate of the first object and the difference between the pixel displacement are smaller than the horizontal tolerance threshold, it is determined that the horizontal condition is met.

8. The method according to claim 7, characterized in that When the first object and the second object are faces, the horizontal tolerance threshold is associated with a pixel distance between left and right eyes.

9. The method according to claim 7, characterized in that When the main camera and the auxiliary camera are horizontally mounted and their optical axes are parallel, the pixel displacement is calculated based on the horizontal resolution of the auxiliary camera, the horizontal field of view, and the lens spacing between the main camera and the auxiliary camera.

10. The method according to claim 7, characterized in that When the main camera and the auxiliary camera are mounted horizontally and the optical axis has an inclination angle, the pixel displacement is calculated based on the horizontal resolution of the auxiliary camera, the horizontal field of view, the inclination angle, and the lens spacing between the main camera and the auxiliary camera.

11. The method according to claim 7, characterized in that When the main camera and the auxiliary camera are horizontally mounted and the optical axis has an outward angle, the pixel displacement is calculated based on the horizontal resolution of the auxiliary camera, the horizontal field of view, the outward angle, and the lens spacing between the main camera and the auxiliary camera.

12. The method according to claim 1, characterized in that There is a first identifiable object in the physical scene, the main camera can perform depth acquisition on the first identifiable object, and the auxiliary camera cannot perform depth acquisition on the first identifiable object.

13. The method according to claim 1, characterized in that The physical scene has a first identifiable object. The main camera and the auxiliary camera can both acquire depth of the first identifiable object. The depth of the first identifiable object relative to the main camera is greater than or equal to the depth of the first identifiable object relative to each of the auxiliary cameras.

14. The method according to claim 1, wherein: The number of the auxiliary cameras is greater than 1, the main camera and the auxiliary camera have a plurality of overlapping fields of view, wherein the sum of the distances between the main camera and each of the overlapping fields of view is less than the sum of the distances between any of the auxiliary cameras and each of the overlapping fields of view.

15. The method according to claim 1, wherein: The number of the auxiliary cameras is greater than 1, the main camera and the auxiliary cameras have multiple overlapping fields of view, wherein the physical scene has multiple identification objects, wherein the total number of pixel coordinate transformations required for the main camera relative to the identification objects is less than the total number of pixel coordinate transformations required for any of the auxiliary cameras relative to each of the identification objects.

16. The method according to claim 1, wherein: The main camera and the auxiliary camera are mounted horizontally, wherein the depth detection limit of the main camera is a first depth value, and when the lens distance between the main camera and the auxiliary camera is greater than a preset distance, the depth detection limit of the main camera and the auxiliary camera is a second depth value, wherein the second depth value is less than the first depth value, wherein the preset distance is associated with the first depth value, the second depth value and the horizontal field of view of the main camera.

17. A processing device connected to a main camera and at least one auxiliary camera, characterized in that: include: Memory for storing data; as well as Processor for: Receiving a main image and at least one auxiliary image respectively generated by performing image acquisition on a physical scene from the main camera and the auxiliary camera; Performing image superposition on the main image and the auxiliary image to generate a superimposed image; Calculating the position information and depth information of each object in the superimposed image; as well as According to the position information and the depth information of each of the objects, it is determined whether the objects in the superimposed image are the same object. The step of calculating the position information and the depth information of each object in the superimposed image includes: recognizing each of the objects from the superimposed image, wherein the superimposed image includes a primary area corresponding to the primary camera and a secondary area corresponding to the secondary camera; and Determine the vertical coordinates, horizontal coordinates and depth of each of the objects located in the main area and the auxiliary area, The object includes a first object located in the main area, and the step of determining whether the objects in the superimposed image are identical objects according to the position information and the depth information of each object includes: Determine whether the secondary region has a second object whose vertical coordinate, depth, and horizontal coordinate meet the vertical condition, depth condition, and horizontal condition respectively with the vertical coordinate, the depth, and the horizontal coordinate of the first object; When the vertical condition, the depth condition, and the horizontal condition are all met, determining that the objects in the superimposed image are identical objects; and When any one of the vertical condition, the depth condition, and the horizontal condition is not satisfied, it is determined that the objects in the superimposed image do not have the same object.

18. An object tracking system, characterized in that: include: A main camera, used for acquiring images of a physical scene to generate a main image; At least one auxiliary camera, used to acquire an image of the physical scene to generate at least one auxiliary image; as well as A processing device, connected to the main camera and the auxiliary camera, is used to: Acquire the main image and the auxiliary image from the main camera and the auxiliary camera respectively; Performing image superposition on the main image and the auxiliary image to generate a superimposed image; Calculating the position information and depth information of each object in the superimposed image; as well as According to the position information and the depth information of each of the objects, it is determined whether the objects in the superimposed image are the same object. The step of calculating the position information and the depth information of each object in the superimposed image includes: recognizing each of the objects from the superimposed image, wherein the superimposed image includes a primary area corresponding to the primary camera and a secondary area corresponding to the secondary camera; and Determine the vertical coordinates, horizontal coordinates and depth of each of the objects located in the main area and the auxiliary area, The object includes a first object located in the main area, and the step of determining whether the objects in the superimposed image are identical objects according to the position information and the depth information of each object includes: Determine whether the secondary region has a second object whose vertical coordinate, depth, and horizontal coordinate meet the vertical condition, depth condition, and horizontal condition respectively with the vertical coordinate, the depth, and the horizontal coordinate of the first object; When the vertical condition, the depth condition, and the horizontal condition are all met, determining that the objects in the superimposed image are identical objects; and When any one of the vertical condition, the depth condition, and the horizontal condition is not satisfied, it is determined that the objects in the superimposed image do not have the same object.

Images