Apparatus and method for tracking moving object

Abstract

This tracking apparatus is configured to track a recognized object moving in a target space by using information received from a plurality of depth cameras installed in the target space. The tracking apparatus comprises: a camera computer that extracts regions of interest respectively from a plurality of depth maps received from the plurality of depth cameras and combines the extracted regions of interest to generate a full depth map image for the target space and the recognized object; a server computer that generates, from the full depth map image received from the camera computer, a blob data map image including location information about the recognized object within the target space; and a display computer configured to display a tracking image in the target space on the basis of the blob data map image received in real time from the server computer and perform calibration for the difference in position between the recognized object and the tracking image.

Description

Device and method for tracking a moving object

[0001] The present disclosure relates to a device and method for tracking an object moving in a predetermined space in real time.

[0002] Implementing interactive content in large spaces requires accurate tracking of moving objects within that space. Depth cameras and sensors, due to their limited field of view (FOV), can only sense narrow spaces. Consequently, there is no known effective method for real-time aggregation and analysis of data generated from multiple depth cameras or sensors to enable sensing of large spaces using depth cameras. Furthermore, analyzing the aggregated data requires excessive computation, necessitating a tracking technology capable of covering large spaces in real time.

[0003] The matters described in the technical background of this invention are written to enhance understanding of the background of the invention and may include matters that are not already known in the field to which this technology belongs.

[0004] The problem that the present invention seeks to solve is to provide a method capable of accurately tracking an object moving in a large space.

[0005] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0006] A tracking device according to an embodiment of the present invention is configured to track a recognition object moving in a target space using information received from a plurality of depth cameras installed in the target space. The tracking device includes a camera computer which extracts a region of interest from a plurality of depth maps received from the plurality of depth cameras and combines the extracted regions of interest to generate a full depth map image for the target space and the recognition object; a server computer which generates a blob data map image including positional information of the recognition object within the target space from the full depth map image received from the camera computer; and a display computer which is configured to display a tracking image in the target space based on the blob data map image received in real time from the server computer and to perform calibration for a positional difference between the recognition object and the tracking image.

[0007] The camera computer may be configured to extract the region of interest by setting editing points in the plurality of depth maps received from the plurality of depth cameras, and to map the extracted region of interest to the area in charge of the plurality of depth cameras to generate the entire depth map image.

[0008] The above entire depth map image can be transmitted to the server computer in an image type.

[0009] The above entire depth map image can be transmitted to the server computer via a capture card.

[0010] The server computer may be configured to convert the entire depth map image received from the camera computer into a grayscale image and then generate the blob data map image including blob data representing the recognized object based on the size of a blob included in the grayscale image.

[0011] The server computer can generate the blob data map image by removing blobs having a size smaller than a preset size among the blobs included in the grayscale image.

[0012] The above blob data map image can be transmitted to the display computer via real-time streaming.

[0013] The display computer may display the block data map image received through the real-time streaming on a projection mapping for driving a video projector so that the tracking image is displayed in the target space, and the display computer and the camera computer may be configured to perform calibration to reduce the positional difference between the tracking image and the recognized object.

[0014] The above calibration can be accomplished by adjusting the settings for extracting the region of interest.

[0015] A tracking method for tracking a moving recognition object in a target space using information received from a plurality of depth cameras installed in the target space according to an embodiment of the present invention includes the steps of: extracting a region of interest from each of a plurality of depth maps received from the plurality of depth cameras and combining the extracted regions of interest to generate a full depth map image for the target space and the recognition object; generating a blob data map image including position information of the recognition object in the target space from the full depth map image; displaying a tracking image in the target space based on the blob data map image received in real time; and performing calibration to reduce a difference between a position of the recognition object and a position of the tracking image.

[0016] The above entire depth map image can be generated by extracting the region of interest from the plurality of depth maps received from the plurality of depth cameras by setting editing points and mapping the extracted region of interest to the area in charge of the plurality of depth cameras.

[0017] The above blob data map image may include blob data representing the recognized object based on the size of a blob included in the grayscale image after converting the entire depth map image into a grayscale image.

[0018] The step of displaying the tracking image in the target space may display the block data map image received through the real-time streaming on projection mapping for driving a video projector so that the tracking image is displayed in the target space, and the step of performing the calibration may reduce the difference in position between the tracking image and the recognized object.

[0019] The above calibration can be accomplished by adjusting the settings for extracting the region of interest.

[0020] According to the present invention, it is possible to accurately track a recognition object moving in a target space.

[0021] In addition, various effects that can be obtained or expected due to embodiments of the present invention are disclosed directly or implicitly in the detailed description of the embodiments of the present invention.

[0022] The accompanying drawings, which are intended to aid in understanding the present invention, provide embodiments of the present invention along with a detailed description. However, the technical features of the present invention are not limited to any specific drawings, and the features disclosed in each drawing may be combined to form new embodiments. The embodiments of the present specification may be better understood by referring to the following description in conjunction with the accompanying drawings, in which similar reference numerals designate identical or functionally similar elements.

[0023] FIG. 1 is a schematic diagram of a device for tracking a moving object according to an embodiment of the present invention.

[0024] FIG. 2 is a drawing for explaining a tracking process performed by a device for tracking a moving object according to an embodiment of the present invention.

[0025] FIG. 3 is a flowchart of a method for tracking an object moving in a target space according to an embodiment of the present invention.

[0026] It should be understood that the drawings referenced above are not necessarily drawn to scale and are intended to provide brief representations of various features that illustrate the fundamental principles of the present invention. For example, specific design features of the present invention, including specific dimensions, orientations, positions, and shapes, will be determined in part by the specific intended application and usage environment.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the described embodiments.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the terms "comprises" and / or "comprising," as used herein, indicate the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. The term "coupled" indicates a physical relationship between two components in which the components are directly connected to one another or are indirectly connected through one or more intervening components.

[0029] In describing the components of the present invention, when it is described that a component is “connected,” “coupled,” or “connected” to another component, it should be understood that the component may be directly connected, coupled, or connected to the other component, but another component may also be “connected,” “coupled,” or “connected” between each component.

[0030] FIG. 1 is a schematic diagram illustrating a device for tracking a moving object (hereinafter referred to as a "tracking device") according to an embodiment of the present invention. Referring to FIG. 1, the tracking device includes a plurality of depth cameras (11), a camera computer (13), a server computer (15), and a display computer (17).

[0031] A plurality of depth cameras (11) are installed on the ceiling of the space so as to generate a depth map of the target space and cover the entire target space to be tracked. In response to the pixels of the target resolution, for example, FHD (1920 x 1080), the streams of each depth camera (11) can be allocated to a specific area (for example, 240 x 320) of pixels, corresponding to the number of depth cameras (11). For example, when 12 depth cameras are set in a 6 x 2 array with a resolution of 240 x 320, they can be allocated to pixels of 1440 x 640.

[0032] Depth data acquired by the depth camera (11) is transmitted to the camera computer (13). The camera computer (13) generates a depth map for each depth camera (11) and combines the generated depth maps to generate an entire depth map image. The camera computer (13) is configured to generate an entire depth map image through installed software.

[0033] The left part of Fig. 2 shows the entire depth map image (101) displayed on the camera computer (13) for generating the entire depth map image and the depth map (103) of each depth camera (11). The depth map is extracted for each depth camera (11). At this time, the depth map extraction can be performed by applying depth data including the depth distance and a background subtraction algorithm. For the depth map (103) for each depth camera (11), a region of interest (ROI) is specified and extracted for each depth camera (11) through an editing interface (105) provided by the editor program. The ROI extracted for each depth camera (11) is mapped to the region that the corresponding depth camera (11) is responsible for. In this way, a full depth map image can be generated by extracting ROI areas for depth data from all depth cameras (11) and mapping the extracted ROIs to the corresponding areas. Extracting ROIs and combining the extracted ROI areas to generate a full depth map image can be called stitching.

[0034] In this way, the entire depth map image generated by the camera computer (13) is transmitted to the server computer (13). At this time, the entire depth map can be transmitted to the server computer (15) in the form of an image via a capture card. That is, as shown in the middle part of Fig. 2, the display screen image (107) displayed on the display of the camera computer (13) is transmitted to the server computer (15). The display screen image (107) includes the entire depth map image (101) generated by the camera computer (13). Since the entire depth map image (101) is transmitted to the server computer (15) in the form of an image via the capture card, the data transmission load is greatly reduced.

[0035] The server computer (15) separates the entire depth map image (101) from the received display screen image (107), and converts the separated entire depth map image (101) into a grayscale depth map image by binarizing it. In the entire depth map image (101) of Fig. 2, the background portion may represent the floor of the target space, and the portion indicated by the stain may be assumed to represent a moving object, for example, a person. In the converted grayscale depth map image, the background portion may be displayed in black, and the stain portion may be displayed in white.

[0036] Then, the server computer (15) removes noise from the gray-scale depth map image to generate a blob data map. For example, noise can be determined based on the size of a white spot in the gray-scale depth map image, and then the spot determined to be noise can be converted to black to remove the noise. For example, if the size of a white spot in the gray-scale depth map image is smaller than a set size, the blob data map can be generated by determining it as noise and removing the spot. The lower part of the middle part of Fig. 2 shows an image in which block data and noise are divided and displayed according to the size of the spot in the gray-scale depth map.

[0037] The generated blob data map is transmitted to a display computer (17) via NDI (network device interface) streaming. The display computer (17) is configured to drive a video display device, for example, a beam projector (19), which displays an image, for example, an interactive content image, in each area. The display computer (17) can display the blob data map on projection mapping for driving the beam projector (19). Accordingly, as shown in the right part of Fig. 2, tracking images (111, 113) can be displayed by the beam projector (19). The tracking images (111, 13) are portions representing objects of the blob data map. At this time, the tracking image indicated by 111 corresponds to an object (115) in the target space, and the tracking image indicated by 113 corresponds to another object (117). In this case, the position of the tracking image (111) is different from the actual position of the corresponding object (115), and the position of the tracking image (113) matches the actual position of the corresponding object (117).

[0038] In an embodiment of the present invention, calibration is performed to correct the positional difference between a target object (115) and a corresponding tracking image (111). That is, a blob data map obtained by an object moving in the target space is received in real time via NDI streaming, the received blob data map is displayed as an image in the target space by projection mapping, and calibration is performed to eliminate the positional difference between the displayed image and the moving object. At this time, the calibration can be obtained by adjusting the stitching described above.

[0039] Figure 3 is a flowchart of a tracking method for tracking an object moving in a target space according to an embodiment of the present invention. The tracking method can be performed by a tracking device including a camera computer (13), a server computer (15), and a display computer (17).

[0040] The camera computer (13) receives depth camera data from a plurality of depth cameras (11) installed in the target space (S11), and extracts a depth map for each depth camera (11) from the received depth camera data (S12). Then, the camera computer (13) extracts a ROI area through an editing point of the depth map for each depth camera (11) (S13). Then, the camera computer (13) maps the extracted ROI area to the responsible area for each camera to generate an entire depth map image (S14), and transmits the generated entire depth map image (S15).

[0041] The server computer (15) receives the entire depth map in video format (S21). At this time, the server computer (15) can receive the entire depth map in video format through a capture card. Then, the server computer (15) generates a blob data map for a recognized object from the entire depth map (S22) and transmits the generated blob data map to the display computer (17) in real time through NDI streaming.

[0042] The display computer (17) displays the blob data map received via NDI streaming on the projection mapping of a video projector, for example, a beam projector (19), so that it is displayed as a tracking image in the target space (S31). Then, the display computer (17) and the camera computer (13) perform calibration for position adjustment of the projection image based on NDI streaming and projection mapping. At this time, the calibration can be performed by adjusting the settings of the editing point for extracting the ROI from each depth map.

[0043] The above-described tracking device and method can accurately track the movement of a recognized object moving in a target space in real time. Based on this tracking of moving objects, various interactive content can be implemented.

[0044] Although the embodiments of the present invention have been described above, the scope of the present invention is not limited thereto, and includes all changes and modifications that can be easily modified by a person having ordinary skill in the art to which the present invention pertains and are recognized as equivalent from the embodiments of the present invention.

Claims

1. A tracking device configured to track a moving recognition object in a target space by using information received from a plurality of depth cameras installed in the target space, A camera computer for extracting regions of interest from a plurality of depth maps received from the plurality of depth cameras and combining the extracted regions of interest to generate an entire depth map image for the target space and the recognized object; A server computer that generates a blob data map image including location information of the recognized object within the target space from the entire depth map image received from the camera computer; and A tracking device including a display computer configured to display a tracking image in the target space based on the blob data map image received in real time from the server computer and to perform calibration for the difference in position between the recognition object and the tracking image.

2. In paragraph 1, A tracking device configured such that the camera computer extracts the region of interest by setting editing points in the plurality of depth maps received from the plurality of depth cameras, and maps the extracted region of interest to the areas in charge of the plurality of depth cameras to generate the entire depth map image.

3. In paragraph 1, A tracking device in which the entire depth map image is transmitted to the server computer in the form of an image.

4. In paragraph 3, A tracking device in which the entire depth map image is transmitted to the server computer via a capture card.

5. In paragraph 1, A tracking device configured such that the server computer can convert the entire depth map image received from the camera computer into a gray scale image and then generate the blob data map image including blob data representing the recognized object based on the size of a blob included in the gray scale image.

6. In paragraph 5, The above server computer is a tracking device that generates the blob data map image by removing blobs having a size smaller than a preset size among the blobs included in the gray scale image.

7. In paragraph 1, The above blob data map image is a tracking device transmitted to the display computer via real-time streaming.

8. In paragraph 7, The above display computer displays the block data map image received through the real-time streaming on projection mapping to drive a video projector so that the tracking image is displayed in the target space. A tracking device in which the display computer and the camera computer are configured to perform calibration to reduce the difference in position between the tracking image and the recognized object.

9. In paragraph 8, A tracking device in which the above calibration is performed by adjusting the settings for extracting the above region of interest.

10. A tracking method for tracking a recognition object moving in a target space by using information received from a plurality of depth cameras installed in the target space, A step of extracting a region of interest from each of a plurality of depth maps received from the plurality of depth cameras and combining the extracted regions of interest to generate a full depth map image for the target space and the recognized object; A step of generating a blob data map image including location information of the recognized object within the target space from the entire depth map image; A step of displaying a tracking image in the target space based on the blob data map image received in real time; and A tracking method comprising a step of performing calibration to reduce the difference between the position of the recognized object and the position of the tracking image.

11. In paragraph 10, A tracking method in which the entire depth map image is generated by extracting the region of interest by setting editing points in the plurality of depth maps received from the plurality of depth cameras and mapping the extracted region of interest to the area in charge of the plurality of depth cameras.

12. In paragraph 10, A tracking method in which the above blob data map image includes blob data representing the recognized object based on the size of a blob included in the gray scale image after converting the entire depth map image into a gray scale image.

13. In paragraph 10, The step of displaying the above tracking image in the target space is to display the block data map image received through the real-time streaming on projection mapping for driving a video projector so that the tracking image is displayed in the target space. A tracking method in which the step of performing the above calibration reduces the difference in position between the tracking image and the recognized object.

14. In paragraph 13, The above calibration is a tracking method that is performed by adjusting the settings for extracting the above region of interest.

Images