Object tracking device, object tracking method, and program

Abstract

This object tracking device includes a processor that executes processing of acquiring a relationship between an object bbox coordinate group at time T-1 and an object bbox coordinate group at time T by matrix calculation, and associating the object bbox coordinate group at the time T-1 with the object bbox coordinate group at the time T on the basis of the acquired relationship represented by a matrix.

Description

Object tracking device, object tracking method, and program

[0001] An embodiment of the present invention relates to an object tracking device, an object tracking method, and a program.

[0002] There are various techniques for tracking a subject in a video and assigning a unique identifier to the same subject (see, for example, Non-Patent Documents 1 and 2).

[0003] Such subject tracking technology is essential when building systems that collect and analyze subject information in a chronological order, such as systems that can accurately detect people behaving suspiciously or who appear to be unwell and are caught on surveillance cameras at event venues, etc.

[0004] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu & Xinggang Wang. ByteTrack: Multi-object Tracking by Associating Every Detection Box. in Proceedings of the European Conference on Computer Vision (ECCV), 2022. Yuang Zhang, Tiancai Wang, Xiangyu Zhang. MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors. in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22056-22065, 2023.

[0005] In order to track a large number of subjects using a large-scale model based on deep learning technology, it is essential to make maximum use of the processing power of the GPU (Graphics Processor Unit) used in the system.

[0006] However, the technical algorithms for tracking objects disclosed in the prior art documents have a structure in which processing that is not suitable for a GPU, such as for loop processing and if branching processing, is essential in the tracking process. Therefore, as the number of objects increases, the load when processing the processing on the CPU (Central Processing Unit) or GPU becomes a bottleneck, and the processing capabilities of the GPU cannot be fully utilized. This can sometimes limit the upper limit of the number of objects that can be tracked per unit time.

[0007] Therefore, it is desirable to eliminate the processing that is not suitable for the GPU and that has become a bottleneck in processing.

[0008] The present invention has been made in consideration of these problems, and aims to provide an object tracking device, an object tracking method, and a program that can reduce processing that is not suitable for a GPU, suppress processing bottlenecks on the CPU or GPU, and enable tracking of more subjects per unit time.

[0009] The object tracking device according to the present invention includes a processor that executes processing to obtain a relationship between an object bbox coordinate group at time T-1 and an object bbox coordinate group at time T by matrix calculation, and to associate the object bbox coordinate group at time T-1 with the object bbox coordinate group at time T based on the obtained relationship expressed as a matrix.

[0010] According to the present invention, it is possible to reduce processing that is not suitable for the GPU, suppress processing bottlenecks on the CPU and GPU, and track more subjects per unit time.

[0011] FIG. 1 is a block diagram showing an example of the functional configuration of an object tracking device 10 according to the first embodiment. FIG. 2 is a flowchart showing an example of the operation of the information fusion unit 11 of the object tracking device 10. FIG. 3 is a flowchart showing an example of the operation of the image feature quantity generation unit 12 of the object tracking device 10. FIG. 4 is a flowchart showing an example of the operation of the tracking feature quantity generation unit 13 of the object tracking device 10. FIG. 5 is a flowchart showing an example of the operation of the tracking matrix generation unit 15 of the object tracking device 10. FIG. 6 is a diagram outlining the process of generating a tracking matrix Mt(T) at time T from feature quantity relationship maps MpL1 to MpLN at time T according to the processing of the tracking matrix generation unit 15. FIG. 7 is a diagram conceptually illustrating the process of generating a tracking matrix Mt(T) at time T from feature quantity relationship maps MpL1 to MpLN at time T according to the processing of the tracking matrix generation unit 15. FIG. 8 is a flowchart showing an example of the operation of the identification calculation unit 16 of the object tracking device 10. FIG. 9 is a diagram conceptually illustrating the process of associating an object bbox coordinate group at time T−1 with an object bbox coordinate group at time T according to processing by the identification calculation unit 16. FIG. 10 is a block diagram illustrating an example (part 1) of the hardware configuration of the object tracking device 10 according to the first embodiment. FIG. 11 is a diagram outlining the differences between conventional object tracking processing according to the prior art and object tracking processing by the object tracking device 10 (SV1) according to the first embodiment, where FIG. 11A is a diagram outlining the hardware load sharing of the conventional object tracking processing, and FIG. 11B is a diagram outlining the hardware load sharing of the object tracking processing according to the first embodiment. FIG. 12 is a diagram illustrating a comparison of the processing load in conventional object tracking processing according to the prior art and the processing load in the object tracking processing by the object tracking device 10 (SV1) according to the first embodiment. FIG. 13 is a block diagram illustrating an example (part 2) of the hardware configuration of the object tracking device 10 according to the first embodiment. Fig. 14 is a diagram outlining load sharing on the hardware of the object tracking device 10 shown in Fig. 13. Fig. 15 is a block diagram showing an example of the functional configuration of the object tracking device 20 according to the second embodiment. Fig. 16 is a flowchart showing an example of the operation of the identification calculation unit 17 of the object tracking device 20. Fig. 17 is a block diagram showing an example of the hardware configuration of the object tracking device 20 according to the second embodiment.FIG. 18 is a diagram outlining the load sharing on the hardware of the object tracking device 20 shown in FIG.

[0012] Hereinafter, an embodiment of an object tracking device, an object tracking method, and a program according to the present invention will be described with reference to the drawings.

[0013] The configurations, numerical values, processing flows, functional elements, and the like described in the following embodiments are merely examples, and the technical scope of the present disclosure is not limited to the following descriptions. The division of the components constituting each of the following embodiments (e.g., division by functional unit) is one example in which the embodiment can be realized. The configurations that can realize each embodiment are not limited to the following examples, and various configurations are possible. The components constituting each of the following embodiments may be further divided, and one or more components constituting each of the following embodiments may be integrated. Furthermore, each device or unit can be combined as appropriate within a consistent range.

[0014] (Configuration of First Embodiment) FIG. 1 is a block diagram showing an example of the functional configuration of an object tracking device 10 according to the first embodiment.

[0015] The object tracking device 10 is realized by having the processor (CPU: Central Processing Unit and GPU: Graphics Processor Unit) execute the functions of each part shown in FIG. 1 in accordance with a program in an information processing device such as a server device, a PC (Personal Computer), or a tablet terminal.

[0016] The object tracking device 10 is a device that receives as input, for example, a frame image captured by a camera at time T, an object bbox coordinate group (bbox: bounding box) detected by an object detector such as YOLOX (Exceeding You Only Look Once), and tracking features at time T-1, and outputs an object bbox coordinate group at time T associated with the object bbox coordinate group at time T-1 and the tracking features.

[0017] The object tracking device 10 includes an information fusion unit 11 , an image feature generation unit 12 , a tracking feature generation unit 13 , a feature relationship map storage unit 14 , a tracking matrix generation unit 15 , and an identification calculation unit 16 .

[0018] (Operation of First Embodiment) The operation of each unit involved in the inference process of the object tracking device 10 will be specifically described.

[0019] FIG. 2 is a flowchart showing an example of the operation of the information fusion unit 11 of the object tracking device 10.

[0020] First, the information fusion unit 11 acquires a group of object bbox coordinates at time T from outside the object tracking device 10 (step S11).

[0021] The object bbox may be any information indicating the area of ​​the object. For example, it may be a rectangular area output by an object detection model such as YOLOX, that is, a set of x and y coordinates representing the center of the rectangle and the confidence of the bbox prediction (x t ,y t ,s t ) is possible. Other coordinate systems that represent shapes such as ellipses or polygons are also possible. An object bbox coordinate group is a batch of multiple object bbox coordinate information items organized into a single data set using an array or the like.

[0022] Next, the information fusion unit 11 acquires tracking features at time T-1 from outside the object tracking device 10 (step S12). Note that when time T=0, the process of step S12 is omitted.

[0023] Next, the information fusion unit 11 generates a detection feature at time T from the information on the object bbox coordinate group at time T and the learning query (step S13). In the process of step S13, the detection feature at time T may be anything as long as it can be generated from the information on the object bbox coordinate group at time T and the learning query. For example, as described in Non-Patent Document 2, if the number of proposals by YOLOX is N t ,When the number of dimensions of the learning query is D, we divide the 1 × D learning query into N t ×D, and the confidence s of the object bbox coordinates tcan be generated by embedding using sine-cosine positional encoding and summing it with the broadcasted training query.

[0024] Next, the information fusion unit 11 generates a fusion feature at time T by combining the detection feature at time T generated in step S13 and the tracking feature at time T-1 (step S14). Note that when time T=0, step S14 is omitted and the detection feature is used as the fusion feature.

[0025] Next, the information fusion unit 11 outputs the fusion feature at time T to the tracking feature generation unit 13 (step S15).

[0026] FIG. 3 is a flowchart showing an example of the operation of the image feature amount generating unit 12 of the object tracking device 10.

[0027] First, the image feature amount generating unit 12 acquires a frame image at time T from outside the object tracking device 10 (step S21).

[0028] Next, the image feature generator 12 generates image features of the frame image at time T (step S22). The process of step S22 may be any process that converts the frame image at time T into image features. For example, as shown in Non-Patent Document 2, image features can be generated by conversion using ResNet-50, which is a convolutional neural network, and a Deformable End-to-End Object Detection with Transformers (DETR) Encoder.

[0029] Next, the image feature generator 12 outputs the image feature of the frame image at time T to the tracking feature generator 13 (step S23).

[0030] FIG. 4 is a flowchart showing an example of the operation of the tracking feature generator 13 of the object tracking device 10.

[0031] The tracking feature generator 13 first acquires an object bbox coordinate group at time T and an object bbox coordinate group at time T-1 from outside the object tracking device 10, and acquires a combined bbox coordinate group by combining them (step S31).

[0032] Next, the tracking feature generator 13 acquires the fusion feature at time T from the information fusion unit 11 (step S32).

[0033] Next, the tracking feature generator 13 acquires the image feature at time T from the image feature generator 12 (step S33).

[0034] Next, the tracking feature generation unit 13 generates tracking features at time T using the combined bbox coordinate group acquired in steps S31 to S33, the fusion features at time T, and the image features at time T (step S34). As described in Non-Patent Document 2, tracking features at time T can be generated using a Deformable DETR Decoder and a Query Interaction Module (QIM). The processing of step S34 is not limited to Deformable DETR and QIM, and any processing may be used as long as it generates tracking features at time T using the combined bbox coordinate group, the fusion features at time T, and the image features at time T and can consistently implement the tracking feature generation unit 13.

[0035] Next, the tracking feature generation unit 13 acquires a feature relationship map from the generative model of step S34 and outputs it to the feature relationship map storage unit 14 (step S35). For example, a self-attention map of the fused features in each layer of the Deformable DETR can be acquired and output to the feature relationship map storage unit 14. The processing of step S35 is not limited to this, and any mapping information expressing the relationship between the fused features may be used.

[0036] Next, the tracking feature generation unit 13 outputs the tracking feature at time T to the outside of the object tracking device 10 (step S36). Here, the tracking feature at time T output from the object tracking device 10 is acquired by the information fusion unit 11 as the tracking feature at time T-1 when the object tracking device 10 acquires the next frame image at time T.

[0037] The feature amount relation map storage unit 14 stores the feature amount relation map of each layer output from the tracking feature amount generation unit 13 .

[0038] FIG. 5 is a flowchart showing an example of the operation of the tracking matrix generation unit 15 of the object tracking device 10.

[0039] FIG. 6 is a diagram outlining the process of generating the tracking matrix Mt(T) at time T from the feature amount relationship maps MpL1 to MpLN at time T in accordance with the processing of the tracking matrix generation unit 15.

[0040] FIG. 7 is a diagram conceptually showing the process of generating a tracking matrix Mt(T) at time T from the feature amount relationship maps MpL1 to MpLN at time T in accordance with the processing of the tracking matrix generation unit 15.

[0041] Here, the feature quantity relation map storage unit 14 stores all past feature quantity relation maps MpL1 to MpLN of each layer output from the tracking feature quantity generation unit 13.

[0042] The tracking matrix generation unit 15 first acquires the feature quantity relationship maps MpL1 to MpLN of each layer at time T from the feature quantity relationship map storage unit 14 (step S51) (see FIGS. 6A and 7A).

[0043] Next, the tracking matrix generation unit 15 generates a relationship map matrix Mtp(T) at time T that expresses the correspondence between the detection feature (Proposal Q) and the tracking feature (Track Q) from the feature relationship maps MpL1 to MpLN (step S52).

[0044] The number of proposals included in the detected features is N d , the number of proposals included in the tracking features is N tr Then, the feature relationship map of a certain layer is (N d +N tr ) × (N d +N tr ) matrix format. At this time, N d < c1 ≦ N tr columns and 0 ≦ r1 ≦ N d The region indicated by the range of rows is the first partial relation map mp1, 0 ≦ c2 ≦ N d Column and N d < r2 ≦ N trWhen the area indicated by the row range is the second partial relationship map mp2, the first partial relationship map mp1 and the second partial relationship map mp2 of each layer are summed as a matrix and the element values ​​are normalized to generate the relationship map matrix Mtp(T) at time T (see Figures 6(B) and 7(B)).

[0045] The process of generating the relationship map matrix Mtp(T) at time T is an example of implementation, and the present invention is not limited to this process.

[0046] Next, the tracking matrix generation unit 15 uses the relationship map matrix Mtp(T) at time T to generate a tracking matrix Mt(T) that represents matching points between the proposal corresponding to the detected feature at time T and the proposal corresponding to the tracking feature as “1” and other points as “0” (step S53) (see Figures 6(C) and 6(D) and Figure 7(C)).

[0047] 6B, 6C, and 6D, a row-wise tracking matrix Mtr(T) is generated by applying an argmax function to the relationship map matrix Mtp(T) in the row direction, and a column-wise tracking matrix Mtc(T) is generated by applying an argmax function to the relationship map matrix Mtp(T) in the column direction at time T. Then, the product of the elements is calculated to generate a tracking matrix Mt(T) at time T, which is a matrix expressing the correspondence between the objects at time T−1 and time T.

[0048] Next, the tracking matrix generation unit 15 outputs the tracking matrix Mt(T) at time T to the identification calculation unit 16 (step S54).

[0049] FIG. 8 is a flowchart showing an example of the operation of the identification calculation unit 16 of the object tracking device 10.

[0050] FIG. 9 is a diagram conceptually showing the process of associating an object bbox coordinate group at time T−1 with an object bbox coordinate group at time T according to the processing of the identification calculation unit 16. In FIG.

[0051] First, the identification calculation unit 16 acquires a group of object bbox coordinates at time T from outside the object tracking device 10 (step S61).

[0052] Next, the identification calculation unit 16 acquires the tracking matrix Mt(T) at time T output from the tracking matrix generation unit 15 (step S62).

[0053] Next, the identification calculation unit 16 aligns the object bbox coordinate group at time T with the object bbox coordinate group at time T−1 based on the tracking matrix Mt(T) at time T (step S63).

[0054] Note that the processing in step S63 may be any as long as it can associate the object bbox coordinate group at time T with the object bbox coordinate group at time T−1. For example, as shown in FIG. 9 , by taking the product of the object bbox coordinate group at time T and the tracking matrix Mt(T) at time T, it becomes possible to associate the identifier IDn corresponding to each object bbox coordinate group.

[0055] That is, for example, each element value in the relationship map matrix Mtp(T) for time T shown in FIG. 6(B) represents the probability that the object at time T-1 matches the object at time T, and an element value of "1" in the tracking matrix Mt(T) for time T shown in FIG. 6(D) represents that the object at time T-1 matches the object at time T. Therefore, by performing a multiplication process of the object bbox coordinate group for time T based on the tracking matrix Mt(T) for time T, it is possible to associate each object bbox coordinate group at time T-1 with each object bbox coordinate group at time T.

[0056] Next, the identification calculation unit 16 outputs the object bbox coordinate group for time T associated with the object bbox coordinate group for time T-1 from the object tracking device 10 (step S64). Here, the object bbox coordinate group for time T output from the object tracking device 10 is stored, for example, in a storage device (not shown) with an identifier IDn added. The object bbox coordinate group for time T output from the object tracking device 10 is acquired by the tracking feature generation unit 13 as the object bbox coordinate group for time T-1 when the object tracking device 10 acquires the next frame image for time T.

[0057] In this way, in the object tracking device 10 of the first embodiment, the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T is analyzed by matrix calculation in accordance with the processing in the feature amount relationship map storage unit 14, the tracking matrix generation unit 15, and the identification calculation unit 16, and by using a processing algorithm that is completed on the GPU without including processing that is not suitable for a GPU, such as loop processing or branching processing, the processing load on the CPU can be reduced and more subjects can be tracked.

[0058] FIG. 10 is a block diagram showing an example (part 1) of the hardware configuration of the object tracking device 10 according to the first embodiment.

[0059] 10 , the object tracking device 10 according to the first embodiment is configured, for example, by a server computer SV1 or a personal computer, and has a central processing unit (CPU) 101A and a graphics processor unit (GPU) 101B, which are hardware processors. A memory 102, a nonvolatile storage device 103, a drive device 104 for a storage medium R, an input / output interface (I / F) 105, and a network interface (I / F) 106 are connected to the CPU 101A and the GPU 101B via a bus 107.

[0060] The network interface 106 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network N. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0061] The input / output interface 105 is connected to an input device, which is an external device 201 such as a camera and is used by a user, and an output device such as a display, which are attached to the object tracking device 10.

[0062] The input / output interface 105 takes in operation data and voice data input by a user or the like through input devices such as a keyboard, touch panel, touchpad, mouse, and microphone, and outputs the output data to output devices including display devices using liquid crystal or organic electroluminescence (EL) and voice output devices using speakers, for display or voice output. Note that the input and output devices may be devices built into the object tracking device 10, or may be input and output devices of another information processing device (information processing terminal) that can communicate with the object tracking device 10 via the network N.

[0063] The memory 102 is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and stores programs necessary to execute various control processes according to the embodiment.

[0064] The non-volatile storage device 103 is a tangible storage medium that is, for example, a combination of the aforementioned non-volatile memory and a volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and generated during various processes.

[0065] The object tracking device 10 according to the first embodiment can be configured as a data processing device having, as software processing function units, an information fusion unit 11, an image feature generation unit 12, a tracking feature generation unit 13, a feature relationship map storage unit 14, a tracking matrix generation unit 15, and an identification calculation unit 16 shown in FIG. 1.

[0066] Each information storage unit used as a working memory by each unit of object tracking device 10 can be configured using memory 102 or nonvolatile storage device 103 shown in Fig. 10. However, these configured storage areas are not essential components within object tracking device 10, and may be areas provided in, for example, an external storage medium R such as a USB (Universal Serial Bus) memory connected via drive device 104, or a storage device such as a database server located in the cloud.

[0067] The processing function units in each of the aforementioned information fusion unit 11, image feature amount generation unit 12, tracking feature amount generation unit 13, feature amount relationship map storage unit 14, tracking matrix generation unit 15, and discrimination calculation unit 16 can all be realized by having the CPU 101A and GPU 101B, which are hardware processors, read and execute programs stored in the memory 102. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0068] The server computer SV1, which is the object tracking device 10, operates by acquiring frame images to be subjected to object tracking processing from an external device 201 such as a camera, and acquiring object bbox coordinate groups from the frame images.

[0069] 11A and 11B are diagrams outlining the differences between conventional object tracking processing according to the prior art and object tracking processing by the object tracking device 10 (SV1) of the first embodiment, where FIG. 11A is a diagram outlining the load sharing on the hardware of conventional object tracking processing, and FIG. 11B is a diagram outlining the load sharing on the hardware of object tracking processing according to the first embodiment.

[0070] Here, it is assumed that the results of acquiring object bbox coordinate groups from frame images are stored (stored) in the memory on the motherboard of the server computer SV1, which is the object tracking device 10 of the first embodiment.

[0071] FIG. 12 is a diagram showing a comparison between the processing load in a conventional object tracking process according to the prior art and the processing load in an object tracking process by the object tracking device 10 (SV1) of the first embodiment.

[0072] 11(A) and 12(A), in conventional object tracking processing according to the prior art, since the processing includes processes unsuitable for GPUs, such as loop processing and branch processing, the processing is not completed solely on the GPU, but instead goes through processing by the CPU once, resulting in bottlenecks such as CPU processing delays and communication of intermediate data between the CPU and the GPU. Even if middleware optimized for GPU-unsuitable processing is used, it is necessary to execute GPU-unsuitable processing multiple times depending on the number of subjects, and this load creates a bottleneck. As a result, the GPU resources are not fully utilized, and the number of subjects that can be tracked is limited.

[0073] On the other hand, as shown in FIGS. 11B and 12B, in the object tracking device 10 (SV1) of the first embodiment, as described above, the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T is analyzed by matrix calculation, and processing that is not suitable for GPU 101B is eliminated. This reduces bottlenecks such as processing delays by CPU 101A and communication of intermediate data between CPU 101A and GPU 101B, or bottlenecks in GPU 101B itself, making it possible to track more subjects per unit time.

[0074] In the hardware configuration of object tracking device 10 shown in FIG. 10, server computer SV1, which is object tracking device 10, is equipped with CPU 101A and GPU 101B, and is configured to perform a series of object tracking processes including receiving frame images and obtaining object bbox coordinate groups. However, as shown in FIGS. 13 and 14, for example, a first server computer SV1-1 equipped with CPU 101A, a second server computer SV1-2 equipped with GPU 101B-1 for object detection, and a third server computer SV1-3 equipped with GPU 101B-2 for object tracking may be communicatively connected via a communication network, and a series of processes may be distributed among each of server computers SV1-1 to SV1-3 to operate as object tracking device 10.

[0075] FIG. 13 is a block diagram showing an example (part 2) of the hardware configuration of the object tracking device 10 according to the first embodiment.

[0076] 13 illustrates (simplified) only the hardware elements necessary for the explanation herein. The first server computer SV1-1 and the second server computer SV1-2, and the first server computer SV1-1 and the third server computer SV1-3 are connected to each other by NICs (Network Interface Cards) that use InfiniBand to communicate between the memories of GPUs 101B-1 and 101B-2 using RDMA (Remote Direct Memory Access).

[0077] FIG. 14 is a diagram outlining the load sharing on the hardware of the object tracking device 10 shown in FIG.

[0078] 14, CD1, CD2, and CD3 are GPU cards. The first server computer SV1-1 mainly performs processing related to distributed processing such as control of asynchronous processing, the second server computer SV1-2 mainly performs processing related to object detection, and the third server computer SV1-3 mainly performs processing related to object tracking.

[0079] In the object tracking device 10 having such a hardware configuration, the first server computer SV1-1 receives frame images to be subjected to object tracking processing from an external device 201 such as a camera, and stores (stores) the coordinate groups of the received frame images in the GPU memory (CD1). The GPU 101B-1 (CD2) of the second server computer SV1-2 then detects and acquires object bbox coordinate groups from the coordinate groups of the frame images, and returns the object bbox coordinate groups to the first server computer SV1-1. The first server computer SV1-1 transmits the object bbox coordinate group data returned from the second server computer SV1-2 to the third server computer SV1-3, where the GPU 101B-2 (CD3) of the third server computer SV1-3 performs object tracking processing, and returns the tracking results to the first server computer SV1-1.

[0080] In this way, once the frame image to be subjected to the object tracking process can be acquired in the GPU memory (CD1) of the first server computer SV1-1, the third server computer SV1-3 can analyze the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T using matrix operations as described above, thereby offloading the object tracking process to the GPU and enabling tracking of more subjects per unit time.

[0081] (Summary of First Embodiment) According to the object tracking device 10 of the first embodiment, the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T is analyzed by matrix calculation, and by making it a processing algorithm that is completed on the GPU without including processing that is not suitable for the GPU, such as loop processing or branching processing, the processing load on the CPU can be reduced, and more subjects can be tracked.

[0082] Therefore, processing that is not suitable for the GPU can be reduced, suppressing processing bottlenecks on the CPU and GPU, and making it possible to track more subjects per unit time.

[0083] (Configuration of Second Embodiment) FIG. 15 is a block diagram showing an example of the functional configuration of an object tracking device 20 according to the second embodiment.

[0084] As in the first embodiment, the object tracking device 20 is realized in an information processing device such as a server device, a PC (Personal Computer), or a tablet terminal equipped with a processor (CPU: Central Processing Unit and GPU: Graphics Processor Unit) by having the processor execute the functions of each part shown in FIG. 15 in accordance with a program (through cooperation between hardware and software).

[0085] The object tracking device 20 is a device that receives as input, for example, a frame image at time T captured by a camera, an object bbox coordinate group (bbox: bounding box) detected by an object detector such as YOLOX (Exceeding You Only Look Once), an object keypoint coordinate group estimated by a skeleton detection model such as HRNet (High-Resolution Network), and tracking features at time T-1, and outputs an object bbox coordinate group / object keypoint coordinate group at time T that are associated with the object bbox coordinate group / object keypoint coordinate group at time T-1 and the tracking features.

[0086] The object tracking device 20 includes an information fusion unit 11 , an image feature amount generation unit 12 , a tracking feature amount generation unit 13 , a feature amount relationship map storage unit 14 , a tracking matrix generation unit 15 , and an identification calculation unit 17 .

[0087] (Operation of Second Embodiment) The operation of each unit involved in the inference process of the object tracking device 20 will be specifically described.

[0088] The processing of the information fusion unit 11, image feature amount generation unit 12, tracking feature amount generation unit 13, feature amount relationship map storage unit 14, and tracking matrix generation unit 15 is the same as that in the first embodiment, and therefore description thereof will be omitted.

[0089] FIG. 16 is a flowchart showing an example of the operation of the identification calculation unit 17 of the object tracking device 20.

[0090] First, the identification calculation unit 17 acquires a group of object bbox coordinates at time T from outside the object tracking device 20 (step S71).

[0091] Next, the identification calculation unit 17 acquires a group of object keypoint coordinates at time T from outside the object tracking device 20 (step S72).

[0092] Here, the object keypoints are, for example, information indicating the skeleton points of an object when the object is a living organism such as a human, and any information indicating skeleton points may be used. For example, a human skeleton output from a skeleton detection model such as HRNet, that is, a set of x and y coordinates of each skeleton point and a set of confidence scores for bbox prediction (x k,t , y k,t , s k,t ) can be considered.

[0093] The object keypoint coordinate group is a batch of coordinate information of multiple object keypoints arranged into one data set by an array or the like.

[0094] Next, the identification calculation unit 17 acquires the tracking matrix Mt(T) at time T from the tracking matrix generation unit 15 (step S73).

[0095] Next, the identification calculation unit 17 aligns the object bbox coordinate group at time T with the object bbox coordinate group at time T-1 based on the tracking matrix Mt(T) at time T (step S74). The processing in step S74 may be any processing as long as it can associate the object bbox coordinate group at time T with the object bbox coordinate group at time T-1. For example, as shown in FIG. 9 , by taking the product of the object bbox coordinate group at time T and the tracking matrix Mt(T) at time T, it becomes possible to associate the identifier IDn corresponding to each object bbox coordinate group.

[0096] Next, the identification calculation unit 17 aligns the object keypoint coordinate group at time T with the object keypoint coordinate group at time T-1 based on the tracking matrix Mt(T) at time T (step S75). The process in step S75 may be any process as long as it can associate the object keypoint coordinate group at time T with the object keypoint coordinate group at time T-1. For example, the association can be achieved by multiplying the object keypoint coordinate group at time T and the tracking matrix Mt(T) at time T.

[0097] Next, the identification calculation unit 17 outputs the object bbox coordinate group at time T associated with the object bbox coordinate group at time T-1 and the object keypoints coordinate group at time T associated with the object keypoints coordinate group at time T-1 from the object tracking device 20 (step S76). Here, the object bbox coordinate group at time T and the object keypoints coordinate group at time T output from the object tracking device 10 are stored in a storage device (not shown), for example, with an object identifier IDn attached.

[0098] In this way, in the object tracking device 20 of the second embodiment, as in the first embodiment, the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T is analyzed by matrix calculation, and by not including processes that are not suitable for GPUs, such as loop processing or branching processing, the processing load on the CPU is reduced, which not only makes it possible to track more subjects, but also makes it possible to track objects between arbitrary data representations by processing that is suitable for GPUs, by providing the identification calculation unit 17 with an arbitrary data representation that represents an object other than the object bbox coordinate group (here, an object keypoints coordinate group).

[0099] FIG. 17 is a block diagram showing an example of the hardware configuration of the object tracking device 20 according to the second embodiment.

[0100] 17 shows (simplified) only the hardware elements necessary for the explanation here. The first server computer SV1-1 and the second server computer SV1-2, the first server computer SV1-1 and the third server computer SV1-3, and the first server computer SV1-1 and the fourth server computer SV1-4 are each connected using InfiniBand by a NIC (Network Interface Card) capable of communicating between the memories of GPUs 101B-1, 101B-2, and 101B-3 using RDMA (Remote Direct Memory Access).

[0101] FIG. 18 is a diagram outlining the load sharing on the hardware of the object tracking device 20 shown in FIG.

[0102] 18, CD1, CD2, CD3, and CD4 are GPU cards. The first server computer SV1-1 mainly performs processing related to distributed processing such as control of asynchronous processing, the second server computer SV1-2 mainly performs processing related to object detection, the third server computer SV1-3 mainly performs processing related to object tracking, and the fourth server computer SV1-4 mainly performs processing related to skeleton detection.

[0103] In the object tracking device 20 having such a hardware configuration, the first server computer SV1-1 receives frame images to be subjected to object tracking processing from an external device 201 such as a camera, and stores (stores) the coordinate groups of the received frame images in the GPU memory (CD1). Then, the GPU 101B-1 (CD2) of the second server computer SV1-2 detects and acquires object bbox coordinate groups from the coordinate groups of the frame images, and returns them to the first server computer SV1-1. The GPU 101B-3 (CD4) of the fourth server computer SV1-4 detects and acquires object keypoint coordinate groups, and returns them to the first server computer SV1-1. The first server computer SV1-1 transmits the object bbox coordinate group and object keypoint coordinate group data returned from the second server computer SV1-2 and the fourth server computer SV1-4 to the third server computer SV1-3, performs object tracking processing in GPU 101B-2 (CD3) of the third server computer SV1-3, and returns the tracking results to the first server computer SV1-1.

[0104] In this way, once the frame image to be subjected to the object tracking process can be acquired in the GPU memory (CD1) of the first server computer SV1-1, the third server computer SV1-3 can analyze the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T using matrix operations as described above, thereby enabling the object tracking process to be performed by GPU offload for object information other than the object bbox coordinate group (here, the object keypoint coordinate group).

[0105] (Summary of Second Embodiment) According to the object tracking device 20 of the second embodiment, similarly to the first embodiment, the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T is analyzed by matrix calculation, and by reducing the processing load on the CPU, not only does it become possible to track more subjects, but also by providing an arbitrary data representation (here, an object keypoints coordinate group) that represents an object other than the object bbox coordinate group, it becomes possible to track objects between arbitrary data representations using processing suitable for a GPU.

[0106] The techniques described in each of the above-described embodiments can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (a floppy disk, a hard disk, etc.), an optical disc (a CD-ROM, a DVD, an MO, etc.), a semiconductor memory (a ROM, a RAM, a flash memory, etc.), or the like, and can be transmitted and distributed via a communication medium.

[0107] The program stored on the medium also includes a configuration program that configures the software means (including not only the execution program but also tables and data structures) that the computer executes. The computer that implements this device reads the program recorded on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by having its operation controlled by this software means. Note that the recording medium referred to in this specification is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.

[0108] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by selecting and combining the multiple disclosed constituent elements.

[0109] For example, if the problem can be solved and an effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the configuration from which these constituent elements are deleted can be extracted as an invention.

[0110] DESCRIPTION OF SYMBOLS 10...Object tracking device (first embodiment) 20...Object tracking device (second embodiment) 11...Information fusion unit 12...Image feature generation unit 13...Tracking feature generation unit 14...Feature relationship map storage unit 15...Tracking matrix generation unit 16...Discrimination calculation unit (first embodiment) 17...Discrimination calculation unit (second embodiment) MpL1 to MpLN...Feature relationship map (1st layer to Nth layer) Mtp(T)...Relationship map matrix at time T Mt(T)...Tracking matrix at time T SV1...Server computer 101A...CPU 101B...GPU 102...Memory 201...External device such as camera

Claims

1. An object tracking device having a processor that executes the following process: acquires the relationship between an object bbox coordinate group at time T-1 and an object bbox coordinate group at time T through a matrix operation; and associates the object bbox coordinate group at time T-1 with the object bbox coordinate group at time T based on the acquired relationship expressed as a matrix.

2. The object tracking device of claim 1, wherein the processor includes a GPU.

3. An object tracking method in which a processor executes the following process: obtains the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T through a matrix operation; and associates the object bbox coordinate group at time T-1 with the object bbox coordinate group at time T based on the obtained relationship expressed as a matrix.

4. A program that causes a processor to execute the following process: obtain the relationship between the object bbox coordinate group at time T-1 and the object bbox coordinate group at time T through matrix calculation; and associate the object bbox coordinate group at time T-1 with the object bbox coordinate group at time T based on the obtained relationship expressed as a matrix.

Images