Device and method for tracking posture of multiple objects based on single object posture estimator

Abstract

A method, device and non-transitory computer-readable medium for tracking poses of multiple objects. The method includes: detecting an object from each of a plurality of consecutive frames in a video sequence by a processor; estimating the pose of each object in each of the plurality of consecutive frames by the processor; and tracking the pose of each object across the plurality of consecutive frames by the processor.

Description

[0001] Cross-references

[0002] This application claims the benefit of priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application Serial No. 62 / 724,007, filed on August 28, 2018, the entire contents of which are incorporated herein by reference.

[0003] Several references, which may include patents, patent applications, and various publications, are cited and discussed in the description of the present disclosure. Citation and / or discussion of such references is provided solely to clarify the description of the present disclosure and is not an admission that any such reference is "prior art" to the disclosure described herein. All references cited and discussed in this specification are incorporated herein by reference in their entirety to the same extent as if each reference were individually incorporated by reference. Technical Field

[0004] The present disclosure generally relates to the field of computer vision, and more particularly, to a posture tracking device and method for multiple objects based on multiple single-object posture estimators. Background Art

[0005] The background description is provided herein to present the background of the present disclosure in general. Within the scope described in this background technology section, the work of the currently designated inventors and the description of the content that may not be considered as prior art at the time of application are neither explicitly nor implicitly considered as prior art that conflicts with the present disclosure.

[0006] The task of multi-object pose tracking is to estimate the human keypoints of all object instances in a set of images constituting a video sequence and assign a unique ID to each instance-level keypoint across frames in the video sequence. Multi-object pose tracking is one of the most important components in various computer vision applications (e.g., surveillance, sports video analysis).

[0007] Existing pose estimation methods for multiple objects can be divided into top-down and bottom-up methods. On the one hand, top-down methods [8, 16] rely on a detection module to obtain candidates and then apply a single-person human pose estimator to localize human keypoints. On the other hand, bottom-up methods [2, 9, 18, 14] detect human keypoints from all potential candidates and then combine these keypoints into each person's limbs based on various data association techniques. The advantage of bottom-up methods is the excellent trade-off between estimation accuracy and computational cost, as the computational cost of the method is independent of the number of candidates in the image. In contrast, the main advantage of top-down methods is the ability to decompose the task into multiple relatively easy tasks, namely, object detection and single-person pose estimation. Object detectors are good at detecting hard (usually small) candidate objects, so pose estimators will perform better under a concentrated regression space.

[0008] For the task of multi-object pose tracking, accurately estimating the trajectories of human keypoints helps in human action recognition and human interaction understanding. PoseTrack

[12] and ArtTrack

[11] mainly introduce the multi-object pose tracking challenge and propose a graph partitioning formulation to convert the pose tracking problem into a multi-cut problem with minimum cost. However, hand-crafted graph models cannot scale to long and unseen clips. Another research area explores top-down approaches by performing multi-object pose estimation for each frame and linking them based on appearance similarity and temporal adjacency[6,19,20]. One solution is to apply multi-object tracking on human detection candidates across frames and then estimate the human pose for each human tube. Although this is a feasible approach, it ignores the unique properties of keypoints. Compared with the tracked bounding boxes, keypoints can be useful cues for bounding box and keypoint tracking.

[0009] Therefore, an improved method for multi-object pose tracking is needed. Summary of the invention

[0010] In some aspects, the present disclosure relates to a method for tracking the pose of multiple objects. The method may include: detecting an object from each of a plurality of consecutive frames in a video sequence by one or more processors; estimating the pose of each object in each of the plurality of consecutive frames by the one or more processors; and tracking the pose of each object across the plurality of consecutive frames by the one or more processors.

[0011] In some embodiments, the step of detecting an object from each of a plurality of consecutive frames in a video sequence by one or more processors may include: determining, by the one or more processors, a bounding box for each object from each of the plurality of consecutive frames; and determining, by the one or more processors, a confidence score for the bounding box for the object in one of the plurality of consecutive frames.

[0012] In some embodiments, the step of estimating, by one or more processors, the posture of each object in each frame of the multiple consecutive frames may include: determining, by the one or more processors, the key points of the object in one of the multiple consecutive frames from an area defined by the bounding box for the object in one of the multiple consecutive frames; and determining, by the one or more processors, a confidence score for the key points of the object in one of the multiple consecutive frames.

[0013] In some embodiments, the step of estimating, by one or more processors, the pose of each object in each frame of the multiple consecutive frames may include: regressing, by the one or more processors, the area defined by the bounding box for the object in one of the multiple consecutive frames into a heat map of the object in one of the multiple consecutive frames; suppressing, by the one or more processors, the heat map of the object in one of the multiple consecutive frames to key points of the object in one of the multiple consecutive frames using cross-heatmap pose non-maximum suppression (NMS); and determining, by the one or more processors, a confidence score for the key points of the object in one of the multiple consecutive frames.

[0014] In some embodiments, the step of estimating, by one or more processors, the posture of each object in each frame of the multiple consecutive frames includes: applying, by the one or more processors, a model integration mode that combines two or more posture estimation models to determine the key points of the object in one of the multiple consecutive frames, wherein the two or more posture estimation models include at least a first model and a second model.

[0015] In some embodiments, the model integration mode includes an expert mode, in which, for each key point of each object in each frame of the multiple consecutive frames, a weight of 1 is assigned to a pose estimation result with a higher average accuracy AP among the pose estimation results for the first model and the pose estimation results for the second model, and a weight of 0 is assigned to another pose estimation result among the pose estimation results for the first model and the pose estimation results for the second model.

[0016] In some embodiments, the model integration mode includes an averaging mode, in which, for all key points of each object in each frame of the multiple consecutive frames, a weight of 0.5 is assigned to the pose estimation result for the first model, and a weight of 0.5 is assigned to the pose estimation result for the second model.

[0017] In some embodiments, one of the first model and the second model is epoch 291, and the other of the first model and the second model is epoch 293.

[0018] In some embodiments, the step of detecting, by one or more processors, an object from each of a plurality of consecutive frames in a video sequence is implemented by a deformable feature pyramid network (FPN), wherein the deformable feature pyramid network is determined based on an object detector selection mechanism.

[0019] In some embodiments, the step of tracking, by one or more processors, the posture of each object across the multiple consecutive frames may include: assigning, by the one or more processors, the same identifier to the bounding box to indicate the same object among the objects across the multiple consecutive frames; and associating, by the one or more processors, key points within the bounding box with the same identifier across the multiple consecutive frames to establish a posture stream for each object across the multiple consecutive frames.

[0020] In some embodiments, the step of tracking the posture of each object across the multiple consecutive frames by one or more processors may also include: using an adaptive key point trimmer by the one or more processors to adaptively trim the key points of each object in each frame of the multiple consecutive frames.

[0021] In some embodiments, the step of adaptively pruning the key points of each object in each frame of the multiple consecutive frames using an adaptive key point pruner by one or more processors may include: determining, by the one or more processors, a key point discarding threshold for each predefined key point category, wherein each key point of each object in each frame of the multiple consecutive frames belongs to a predefined key point category; discarding, by the one or more processors, key points, wherein a confidence score of the discarded key points is less than a key point discarding threshold for a predefined key point category to which the discarded key points belong; and retaining, by the one or more processors, key points, wherein a confidence score of the retained key points is greater than or equal to the key point discarding threshold for a predefined key point category to which the retained key points belong.

[0022] In some embodiments, the step of detecting an object from each of a plurality of consecutive frames in a video sequence by one or more processors may also include: converting the detection results for each object in each of the plurality of consecutive frames into an openSVAI standardized data format by the one or more processors to generate a standardized detection result.

[0023] In some embodiments, the step of estimating the posture of each object in each frame of the multiple consecutive frames by one or more processors also includes: converting the estimation results for each object in each frame of the multiple consecutive frames into the openSVAI standardized data format by the one or more processors to generate a standardized estimation result.

[0024] In some aspects, the present disclosure relates to a posture tracking device for multiple objects. The device may include: a processor; and a memory having instructions stored thereon, which, when executed by the processor, cause the processor to: detect an object from each of a plurality of consecutive frames in a video sequence; estimate the posture of each object in each of the plurality of consecutive frames; and track the posture of each object across the plurality of consecutive frames.

[0025] In some embodiments, when the instructions are executed by the processor, the processor may also be caused to: determine a bounding box for each of the objects from each of the plurality of consecutive frames; and determine a confidence score for the bounding box for the object in one of the plurality of consecutive frames.

[0026] In some embodiments, when the instruction is executed by the processor, it can also cause the processor to: determine the key points of the object in one of the multiple consecutive frames from the area defined by the bounding box for the object in one of the multiple consecutive frames; and determine the confidence score of the key points of the object in one of the multiple consecutive frames.

[0027] In some embodiments, when the instruction is executed by the processor, it can also cause the processor to: regress the area defined by the bounding box for the object in one of the multiple consecutive frames into a heat map of the object in one of the multiple consecutive frames; use cross-heat map pose non-maximum suppression (NMS) to suppress the heat map of the object in one of the multiple consecutive frames to the key points of the object in one of the multiple consecutive frames; and determine the confidence score of the key points of the object in one of the multiple consecutive frames.

[0028] In some embodiments, when the instruction is executed by the processor, it can also cause the processor to: apply a model integration mode that combines two or more posture estimation models to determine the key points of the object in one of the multiple consecutive frames, wherein the two or more posture estimation models include at least a first model and a second model.

[0029] In some embodiments, the model integration mode includes an expert mode, in which, for each key point of each object in each frame of the multiple consecutive frames, a weight of 1 is assigned to a pose estimation result with a higher average precision (AP) among the pose estimation results for the first model and the pose estimation results for the second model, and a weight of 0 is assigned to another pose estimation result among the pose estimation results for the first model and the pose estimation results for the second model.

[0030] In some embodiments, the model integration mode includes an averaging mode, in which, for all key points of each object in each frame of the multiple consecutive frames, a weight of 0.5 is assigned to the pose estimation result for the first model, and a weight of 0.5 is assigned to the pose estimation result for the second model.

[0031] In some embodiments, one of the first model and the second model is epoch 291, and the other of the first model and the second model is epoch 293.

[0032] In some embodiments, when the instructions are executed by the processor, the processor may also be caused to: assign the same identifier to the bounding box to indicate the same object among the objects spanning the multiple consecutive frames; and associate key points within the bounding box with the same identifier across the multiple consecutive frames to establish a pose flow for each of the objects spanning the multiple consecutive frames.

[0033] In some embodiments, when the instructions are executed by the processor, the processor may be further caused to: use an adaptive key point pruner to adaptively prune key points of each object in each frame of the plurality of consecutive frames.

[0034] In some embodiments, when the instruction is executed by the processor, it can also cause the processor to: determine a key point discarding threshold for each predefined key point category, wherein each key point of each object in each frame of the multiple consecutive frames belongs to a predefined key point category; discard key points, wherein the discarded key points have a confidence score less than the key point discarding threshold for a predefined key point category to which the discarded key points belong; and retain key points, wherein the retained key points have a confidence score greater than or equal to the key point discarding threshold for a predefined key point category to which the retained key points belong.

[0035] In some embodiments, when the instructions are executed by the processor, the processor may also be caused to: convert the detection result of each object in each frame of the plurality of consecutive frames into an openSVAI standardized data format to generate a standardized detection result.

[0036] In some embodiments, when the instructions are executed by the processor, the processor may also be caused to: convert the estimation result for each object in each frame of the plurality of consecutive frames into an openSVAI standardized data format to generate a standardized estimation result.

[0037] In some aspects, the present disclosure relates to a non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor of a computing device, cause the processor to perform any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings illustrate one or more embodiments of the present disclosure and, together with the written description, serve to explain the principles of the present disclosure. Throughout the drawings, the same reference numerals are used wherever possible to refer to the same or similar elements of the embodiments.

[0039] Figure 1A An exemplary block diagram schematically illustrates a configuration structure of a computing device for tracking poses of multiple objects based on multiple single-object pose estimators according to some embodiments of the present disclosure.

[0040] Figure 1B An exemplary block diagram schematically illustrates the configuration structure of a cross-heatmap pose non-maximum suppression (NMS) module according to some embodiments of the present disclosure.

[0041] Figure 1C An exemplary block diagram schematically illustrates a configuration structure of an adaptive key point pruner according to some embodiments of the present disclosure.

[0042] Figures 2A to 2L The figure exemplarily shows the posture estimation process performed by the human posture estimation module according to some embodiments of the present disclosure.

[0043] FIG. 3A to FIG. 3C The output results generated by the candidate detection module, the human body posture estimation module and the human body posture tracking module according to some embodiments of the present disclosure are respectively exemplarily shown.

[0044] Figure 4 A flowchart of a method for tracking postures of multiple objects based on multiple single-object posture estimators according to some embodiments of the present disclosure is schematically shown.

[0045] Figure 5A flowchart of a method for tracking postures of multiple objects based on multiple single-object posture estimators according to other embodiments of the present disclosure is schematically shown.

[0046] Figure 6 A flowchart of a method for suppressing a heat map of an object as a key point using cross-heat map non-maximum (NMS) suppression according to some embodiments of the present disclosure is exemplarily shown.

[0047] Figure 7 The flowchart of a method for pruning key points using an adaptive key point pruner according to some embodiments of the present disclosure is exemplarily shown. DETAILED DESCRIPTION

[0048] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, and many modifications and variations therein will be apparent to those skilled in the art. Various embodiments of the present disclosure are described in detail herein. With reference to the accompanying drawings, the same numerals represent the same components throughout the accompanying drawings. As used in the specification and throughout the claims herein, the meanings of "one", "a kind" and "said" include plural forms, unless the context clearly indicates otherwise. Similarly, as used in the specification and throughout the claims herein, the meaning of "in..." includes "in..." and "on...", unless the context clearly indicates otherwise. In addition, for the convenience of the reader, titles or subtitles may be used in the specification, which will not affect the scope of the present disclosure. In addition, some of the terms used in this specification are more specifically defined below.

[0049] In the context of the present disclosure and in the specific context in which each term is used, the terms used in this specification generally have their ordinary meaning in the art. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide practitioners with additional guidance on the description of the present disclosure. It should be understood that the same thing can be said in more than one way. Therefore, optional language and synonyms can be used for any one or more terms discussed herein, and whether the terms are elaborated or discussed herein does not have any special meaning. Synonyms for certain terms are provided herein, but the listing of one or more synonyms does not exclude the use of other synonyms. Examples used anywhere in this specification, including examples of any terms discussed herein, are exemplary only and in no way limit the scope and meaning of the present disclosure or any exemplary terms. Similarly, the present disclosure is not limited to the various embodiments given in this specification.

[0050] All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs, unless otherwise defined. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meanings in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly rigid manner, unless explicitly defined herein.

[0051] As used herein, "about", "around", "substantially" or "close" shall generally mean within 20 percent, preferably within 10 percent, and more preferably within 5 percent of a given value or range. The numerical values ​​given herein are approximate, meaning that if not explicitly stated, the term "about", "around", "substantially" or "close" can be inferred.

[0052] As used herein, "plurality" means two or more.

[0053] As used herein, the terms "including," "comprising," "having," "involving," etc. should be construed as open-ended, ie, meaning including but not limited to.

[0054] As used herein, the phrase "at least one of A, B, and C" should be construed to mean a logical (A or B or C), using a non-exclusive logical "or." It should be appreciated that one or more steps within a method can be performed in a different order (or concurrently) without altering the principles of the present disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0055] As used herein, the term "module" may include or belong to an application specific integrated circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or a processor group) that executes code, other suitable hardware components that provide the above functions, or a combination of some or all of the above. For example, in a system on a chip, the term "module" may include a memory (shared, dedicated, or a processor group) that stores code executed by a processor.

[0056] As used herein, the term "code" may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. As used above, the term "shared" refers to the ability to use a single (shared) processor to execute some or all of the code from multiple modules. In addition, some or all of the code from multiple modules may be stored by a single (shared) memory. As used above, the term "group" refers to the ability to use a group of processors to execute some or all of the code from a single module. In addition, a group of memories may be used to store some or all of the code from a single module.

[0057] The present disclosure relates to computer systems. As shown, computer components may include physical hardware components (shown as solid boxes) and virtual software components (shown as dashed boxes). It will be appreciated by those skilled in the art that the forms of implementing these computer components may include, but are not limited to, software, firmware, hardware components, or a combination of the above three, unless otherwise indicated.

[0058] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0059] The present disclosure will now be described more fully with reference to the accompanying drawings, in which some embodiments of the present disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, but rather, these embodiments are provided to make the present disclosure more thorough and complete and to fully convey the scope of the present disclosure to those skilled in the art.

[0060] In the following, reference will be made to Figures 1A to 1C , Figures 2A to 2L as well as FIG. 3A to FIG. 3C A specific example of a device for tracking the poses of multiple objects (or people) based on multiple single-object pose estimators according to some embodiments of the present disclosure is described in detail.

[0061] Figure 1A An exemplary block diagram of a configuration structure of a computing device 100 for tracking the posture of multiple objects based on multiple single-object posture estimators according to some embodiments of the present disclosure is schematically shown. In some embodiments, the computing device 100 may be a server computer, a cluster, a cloud computer, a general-purpose computer, a mobile device, a tablet computer, or a dedicated computer. The computing device 100 may detect an object from each of a plurality of consecutive frames in a video sequence; estimate the posture of each object in each of the plurality of consecutive frames; and track the posture of each object across the plurality of consecutive frames. In some embodiments, the computing device 100 may communicate with other computing devices or services. In some embodiments, the communication is performed via a network, which may be a wired network or a wireless network, and may have various forms, such as a public network and a private network.

[0062] like Figure 1AAs shown, the computing device 100 may include, but is not limited to, a processor 112 and a memory 114. In addition, the computing device 100 may also include an optional storage device 116. In some embodiments, the computing device 100 may include other hardware components and software components (not shown) to perform the corresponding tasks of the computing device 100. Examples of these hardware components and software components may include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0063] The processor 112 may be a central processing unit (CPU) configured to control the operation of the computing device 100. The processor 112 may execute an operating system (OS) or other applications for the computing device 100. In some embodiments, the computing device 100 may have more than one CPU as a processor, for example, two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory 114 may be a volatile memory, for example, a random access memory (RAM), for storing data and instructions for the operation of the computing device 100. In some embodiments, the memory 114 may be a volatile memory array. In some embodiments, the computing device 100 may run on more than one memory 114. In addition, the memory 114 may also be a non-volatile memory, such as a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or the like.

[0064] The storage device 116 may be a non-volatile data storage medium for storing an OS (not shown) and other applications for the computing device 100. Examples of the storage device 116 may include non-volatile memory, such as a flash memory, a memory card, a USB drive, a hard drive, a floppy disk, an optical drive, a solid-state drive (SSD), or any other type of data storage device. In some embodiments, the storage device 116 may be local storage, remote storage, or cloud storage. In some embodiments, the computing device 100 may have multiple storage devices 116, which may be the same storage device or different types of storage devices, and applications for the computing device 100 may be stored in one or more storage devices 116 of the computing device 100. In some embodiments, the computing device 100 may be a cloud computer, and the processor 112, the memory 114, and the storage device 116 are shared resources provided on demand through the Internet. In some other embodiments, the computing device 100 may be a workstation communicatively connected to a mainframe server, which stores all data and instructions for operating the computing device 100, and in this case, the storage device 116 may be optionally omitted.

[0065] like Figure 1A As shown, at least one of the image database 150 and the training data 160, and the multi-object posture tracking application 118 may be stored on the storage device 116. However, at least one of the image database 150 and the training data 160, and part or all of the multi-object posture tracking application 118 may be stored in other locations, and the present disclosure is not limited thereto.

[0066] In some embodiments, the multi-object posture tracking application 118 includes a candidate detection module 120, a human posture estimation module 130, and a human posture tracking module 140. In some embodiments, the multi-object posture tracking application 118 may include other applications or modules necessary for the operation of the multi-object posture tracking application 118. It should be noted that each module is implemented by computer executable code or instructions, or data tables or databases, which together constitute an application. In some embodiments, each module may also include sub-modules. In some other embodiments, some modules may be combined into a stack. In some other embodiments, some modules may be implemented as circuits rather than executable codes. In some other embodiments, some or all modules in the multi-object posture tracking application 118 may be located at a remote computing device or distributed in the cloud.

[0067] In some embodiments, the candidate detection module 120, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to detect an object from each of a plurality of consecutive frames in a video sequence. In some embodiments, the candidate detection module 120, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to determine a bounding box for each object from each of a plurality of consecutive frames, and determine a confidence score for the bounding box for the object in one of a plurality of consecutive frames. In this embodiment, the object in each of a plurality of consecutive frames may be a person to be tracked. The video sequence may be stored in the image database 150. However, the present disclosure is not limited thereto.

[0068] The candidate detection module 120 can be implemented by a convolutional neural network (CNN). As an example, the candidate detection module 120 can be implemented by a deformable feature pyramid network (FPN) (e.g., see Lin, TY, Dollar, P., Girshick, RB, He, K., Hariharan, B., Belongie, SJ: Feature pyramid networks for object detection. In: CVPR. vol. 1, p. 3 (2017), the entire contents of which are incorporated herein by reference). In some embodiments, the candidate detection module 120 can be implemented by an FPN network having a ResNet 101 backbone. In other embodiments, the candidate detection module 120 can be implemented by an FPN network having a ResNet 151 backbone. In some embodiments, the candidate detection module 120 can be implemented by an FPN network trained with the ImageNet and COCO datasets. In another embodiment, the candidate detection module 120 can be implemented by an FPN network trained with the CrowdHuman

[17] dataset.

[0069] The above-mentioned deformable FPN network can be determined based on the object detector selection mechanism. Specifically, a pre-trained model from a deformable convolutional neural network (ConvNets) [5] can be used. In order to improve the recall of the candidates, experiments are conducted on the validation sets of PoseTrack2017 [1] and PoseTrack 2018 to select the best object detector. First, since the bounding box location is not provided in the annotation in the PoseTrack2017 dataset, the ground truth bounding box of the candidate is inferred from the annotated key points. In a specific exemplary embodiment, the present disclosure locates the bounding box according to the minimum and maximum coordinates of the 15 key points, and then enlarges the bounding box by 20% in the horizontal and vertical directions. Even if the ground truth bounding box is provided in the PoseTrack 2018 dataset, a more consistent version can be inferred based on the ground truth position of the key points. The inferred ground truth bounding box is used to train the pose estimator.

[0070] For object detectors, a deformable convolutional version of the Region Fully Convolutional Network (R-FCN) [4] and a deformable convolutional version of the Feature Pyramid Network (FPN)

[13] , both with a Residual Net (ResNet) 101 backbone

[10] , are compared. The FPN feature extractor is attached to a Fast Region Convolutional Network (Fast R-CNN) [7] head for detection. The detection results are compared with the ground truth based on the accuracy and recall on the PoseTrack 2017 validation set. To eliminate redundant candidates, candidates with lower likelihood or confidence scores are discarded. Table 1 shows the precision-recall on the PoseTrack 2017 validation set, giving the precision and recall of the detector for various discarding thresholds for bounding boxes. A bounding box is accurate if its intersection over union (IoU) with the ground truth is above a certain threshold (which is set to 0.4 in all experiments). In this case, the deformable FPN achieves a precision of 34.2 and a recall of 83.0, while the deformable R-FCN achieves a precision of 30.3 and a recall of 82.6, clearly indicating that the deformable FPN network performs better in terms of precision and recall. For the PoseTrack2018 validation set, the deformable FPN network also performs better. Therefore, using the above object detector selection mechanism, the deformable FPN network can be selected as a candidate detector to implement the candidate detection module 120.

[0071] Table 1

[0072]

[0073] In addition, experiments were conducted to measure the gap between the ideal detection results and the detection results generated using the selected deformable FPN network. Specifically, based on the fact that the upper limit of the detection is the position of the ground truth bounding box, the ground truth bounding box is input into the subsequent human pose estimation module 130 and the human pose tracking module 140 (described in detail below), and the performance of the ground truth bounding box is compared with the performance of the object detector (i.e., the deformable FPN network) selected based on the validation set. Table 2 shows a comparison of the single-frame pose estimation results using various detectors on the PoseTrack 2017 validation set, and the average accuracy (AP) achieved by ground truth detection, deformable FPN and deformable R-FCN is given for various predefined key points. It can be seen that the performance of pose estimation is about 7% better when using ground truth detection (the total accuracy is 81.7 when using ground truth detection and 74.6 when using deformable FPN). In addition, the pose estimation performance of the Deformable FPN network is 0.9% better than that of the Deformable R-FCN network (74.6 total accuracy for the Deformable FPN network and 73.7 total accuracy for the Deformable R-FCN network), which further verifies that the Deformable FPN network is a suitable choice as an object detector instead of the Deformable R-FCN network.

[0074] Table 2

[0075]

[0076] In addition, as shown in Table 3, a comparison of the multi-frame pose tracking results using various detectors on the PoseTrack 2017 validation set is illustrated, and for each predefined key point, the multi-object tracking accuracy (MOTA) results achieved by ground truth detection, deformable FPN, and deformable R-FCN are given. The term "MOTA" may refer to a criterion for evaluating the performance of a human pose tracking system and / or device. The higher the MOTA result or value, the better the human pose tracking system or device. It can be seen that the performance of pose tracking is about 6% better when using ground truth detection (the MOTA value is 67.0 when using ground truth detection and the MOTA score is 61.3 when using deformable FPN). In addition, the pose tracking performance of the deformable FPN network is 2.3% better than that of the deformable R-FCN network (the MOTA value using the deformable FPN network is 61.3 and the MOTA score using the deformable R-FCN network is 59.0), which further verifies that the deformable FPN network is suitable as an object detector rather than the deformable R-FCN network.

[0077] Table 3

[0078]

[0079] After the deformable FPN network is determined as an object detector based on the object detector selection mechanism and the deformable FPN network is trained using a data set from the training data 160, a video sequence consisting of a plurality of consecutive frames stored in the image database 150 may be acquired / retrieved and input to the candidate detection module 120 to detect an object from each of the plurality of consecutive frames. For example, Figure 3A The output generated by the candidate detection module 120 according to some embodiments of the present disclosure is exemplarily shown. Specifically, Figure 3A As shown, it is detected that each of the plurality of bounding boxes is positioned at a corresponding one of the plurality of objects within a frame.

[0080] Then, the detection result of each object in each frame of the multiple consecutive frames is converted into the openSVAI standardized data format (https: / / doc-opensvai.readthedocs.io / en / latest / ; https: / / media.readthedocs.org / pdf / doc-opensvai / latest / doc-opensvai.pdf, the entire contents of which are incorporated herein by reference) to generate a standardized detection result for each object in each frame of the multiple consecutive frames.

[0081] OpenSVAI is a library that provides modular implementations of components such as object detection, single-person pose estimation, pose tracking, semantic segmentation, etc. Existing open source projects are usually incompatible. In order to build a complex system consisting of multiple components (each component is a self-sufficient research topic), we provide such a library, which also standardizes the input and output of each component. Therefore, the methods of each component are easily replaceable, making it easier to build the system. The input and output are JavaScript Object Notation (JSON) files with standardized data structures, which cover the information required by the vision system: image ID, image path, image name, candidate, candidate ID, candidate pose, candidate mask, etc.

[0082] In some embodiments, when the candidate detection module 120 is loaded into the memory 114 and executed by the processor 112, it can also enable the processor 112 to send the standardized detection results to the human posture estimation module 130 after generating the standardized detection results for each object in each frame of multiple consecutive frames.

[0083] In some embodiments, the human pose estimation module 130, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to estimate the pose of each object in each of the plurality of consecutive frames in response to receiving the standardized detection result for each object in each of the plurality of consecutive frames. In some embodiments, the human pose estimation module 130, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to determine the key points of each object in each of the plurality of consecutive frames from the area defined by the bounding box for each object in each of the plurality of consecutive frames, and determine the confidence score of the key points of each object in each of the plurality of consecutive frames.

[0084] Here, the term "key point" may refer to a set of geometric constraints and interdependencies of the body that are applied by a human body model. In certain embodiments, the key points of a plurality of categories may be defined. Hereinafter, the details of human body posture estimation will be described and explained as examples by adopting 15 classes of key points (e.g., neck key points, left shoulder key points, right shoulder key points, left elbow key points, right elbow key points, left wrist key points, right wrist key points, left knee key points, right knee key points, left ankle key points, right ankle key points, etc.). However, it should be noted that the number of categories of key points is not limited thereto, and more or less categories of key points may be present compared to those described in the example embodiments.

[0085] In some embodiments, the human pose estimation module 130 can be implemented by a convolutional neural network (CNN). In some embodiments, the human pose estimation module 130 can be implemented by a cascaded pyramid network (CPN) (for example, see Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., Sun, J.: Cascaded Pyramid Network for Multi-object Pose Estimation. In: CVPR (2018), the entire contents of which are incorporated herein by reference).

[0086] Specifically, the above CPN network was trained for 260 epochs using the combined dataset of PoseTrack 2018 and COCO, and then fine-tuned for 40 epochs only on the PoseTrack 2018 training set to alleviate the regression of the head. For the COCO dataset, the positions of the bottom and top of the head are not given. These key points can be inferred by roughly interpolating the annotated key points. It has been found that by fine-tuning on the PoseTrack dataset, the prediction of head key points (including neck key points, etc.) can be optimized. During the fine-tuning process, an online hard key point mining technique is used, which only focuses on the loss of the 7 most difficult key points out of the 15 key points.

[0087] After that, the CPN network is tested on the PoseTrack 2017 and PoseTrack 2018 validation sets. That is, both the PoseTrack 2017 and PoseTrack 2018 validation sets are input into the trained CPN network and a set of predicted node coordinates (or keypoint coordinates) are generated in the image coordinate system, and then this set of predicted node coordinates is compared with the ground truth node coordinates to test the pose estimation performance of the trained model. The first two rows of Table 4 show the comparison of single-frame pose estimation results with different ensemble modes on the PoseTrack 2017 validation set, showing that different models (e.g., epoch291 and epoch 293, which are two relatively good training models in terms of average precision (AP)) perform differently on different categories of keypoints. For example, for the PoseTrack 2017 validation set and the PoseTrack 2018 validation set, for epoch 291, the predictions for the shoulder (AP is 81.2) and hip (AP is 72.6) provide better results than the predictions for epoch 293 (AP is 80.8 for the shoulder and AP is 70.1 for the hip). However, epoch 293 performs better on end limbs such as the wrist (AP is 71.3) and ankle (AP is 65.7) than epoch 291 (AP is 70.2 for the wrist and AP is 64.7 for the ankle). Therefore, in some embodiments, in order to make more accurate estimates, an ensemble of two models from epoch 291 and epoch 293 can be used during testing.

[0088] There are two ensemble modes: (1) Average mode, which assigns weight 0.5 to epoch 291 and weight 0.5 to epoch 293 for all predefined keypoint categories; and (2) Expert mode, which assigns weight 1 to the model with higher AP in epoch 291 and epoch 293 for each predefined keypoint category, and weight 0 to the other model. The two ensemble modes are tested and the test results are shown in the last two rows of Table 4. As can be seen from Table 4, the Average mode has an overall AP of 74.8 for all categories of keypoints, while the Expert mode uses the prediction of shoulder / hip from the epoch 291 model and the prediction of end limb from the epoch 293 model, and has an overall AP of 75.0. Both modes outperform epoch 291 (74.6) and epoch 293 (74.6), indicating that both modes perform better than the common test on the pose estimation task. However, it should be noted that although Table 4 only shows the pose estimation results on the PoseTrack 2017 validation set, both the Average mode and the Expert mode show better consistency on both the PoseTrack 2017 and PoseTrack 2018 validation sets.

[0089] Table 4

[0090]

[0091] Therefore, in some embodiments, the model integration mode can be applied to determine the key points of each object in each frame of multiple consecutive frames, which can improve the performance of the human pose estimation module 130. As an example, in the average mode, for all predefined key point categories, a weight of 0.5 is assigned to the first model, and a weight of 0.5 is assigned to the second model for determining the key points of each object in each frame of multiple consecutive frames. As another example, in the expert mode, for each predefined key point category, a weight of 1 is assigned to the model with a higher AP for the corresponding key point category in the first model and the second model, and a weight of 0 is assigned to the other model of the two models to determine the key points of each object in each frame of multiple consecutive frames. Here, it should be noted that the model integration mode is not limited to this, and there may be more than two models for forming the model integration mode. In some embodiments, one of the first model and the second model may be epoch291, and the other may be epoch293. In some embodiments, the average mode and the expert mode are combined, and each category in the first model and the second model is assigned a weight of 0.5:0.5 or 0:1 or 1:0. The weight allocation is based on the corresponding AP. In one example, the weight of the class "head" is assigned to epoch 291 and epoch 293 at a ratio of 0.5:0.5, the weight of the class "hip" is assigned to epoch 291, the weights of the classes "elbow", "wrist", "knee" and "ankle" are assigned to epoch 293, the weight of the class "shoulder" is assigned to epoch 291, or the weights are evenly distributed to epoch 291 and epoch 293.

[0092] After training the CPN network using the data set from the training data 160 and determining to use the set according to the two models epoch 291 and epoch 293 to achieve human pose estimation, the candidate detection module 120 generates a standardized detection result for each object in each frame of the plurality of consecutive frames, which can be input into the human pose estimation module 130 to estimate the pose of each object in each frame of the plurality of consecutive frames. Pose estimation is performed on all bounding boxes (i.e., candidates) from the candidate detection module 120. Specifically, the area defined by the bounding box for each object in each frame of the plurality of consecutive frames is regressed into a heat map of each object in each frame of the plurality of consecutive frames. Then, in order to render more accurate key point locations, for each bounding box, the predicted heat map can be post-processed using cross-heat map pose non-maximum suppression (NMS) (e.g., see Ning, G., Zhang, Z., He, Z.: Knowledge-guided deepfractal neural networks for human pose estimation. IEEE Transactions on Multimedia 20(5), 1246-1259 (2018), the entire contents of which are incorporated herein by reference). In particular, the cross-heat map pose NMS module 132 can be used to suppress the heat map of each object in each frame of multiple consecutive frames to the key points of each object in each frame of multiple consecutive frames.

[0093] For example, Figure 1B Schematically shows an exemplary block diagram of the configuration structure of the cross-heatmap posture NMS module 132 according to some embodiments of the present disclosure. In some embodiments, the cross-heatmap posture NMS module 132 may be included in the human posture estimation module 130. Figure 1BAs shown, the cross-heatmap pose NMS module 132 may include: a heatmap acquisition module 133, a spot collection module 134, a spot suppression module 135, and a heatmap quantity comparison module 136. In some embodiments, when the heatmap acquisition module 133 is loaded into the memory 114 and executed by the processor 112, the processor 112 may receive a set (e.g., N) of heatmaps regressed from the image of the object, where N is a positive integer. In some embodiments, the spot collection module 134, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to collect spots from all N heatmaps for suppression. In some embodiments, the spot suppression module 135, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to determine the spot with the maximum response, suppress other spots from the same heatmap and spots in other heatmaps that are very close to the determined spots in the image coordinate system, and delete the heatmap from the set of heatmaps, and let N=N-1. In some embodiments, the heat map quantity comparison module 136, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to compare the current heat map quantity N=N-1 with 0. This process may be repeated until the current heat map quantity N=N-1 is equal to 0. Thus, the key points of each object in each frame of a plurality of consecutive frames and the corresponding confidence scores of the corresponding key points may be determined.

[0094] In some embodiments, the heatmap NMS process is as follows.

[0095] 1 Detect blobs from all heatmaps.

[0096] 2. Sort all spots based on their response, which is the peak value of that spot.

[0097] 3. Suppress the first-ranked spot (with the largest response): (1) determine the node position corresponding to the spot as the peak position of the spot; (2) suppress other spots from the same heat map and spots in other heat maps that are very close to the determined spot in the image coordinate system, and delete the heat map from the group of heat maps.

[0098] 4. For the remaining spots, continue to sort them and perform suppression on the spots ranked first.

[0099] 5. Repeat step 4 until all spots have been suppressed.

[0100] 6. For heatmaps where the node position has not yet been determined, use its heatmap peak response position.

[0101] The above implementation is open source here: https: / / github.com / Guanghan / GNet-pose / blob / master / testing / utils / utils_n ms.py, the entire contents of which are incorporated into this paper by reference.

[0102] Please note that keypoints correspond to single pixels in the heatmap, while blobs are obtained from the heatmap via blob detection. A blob is a collection of adjacent pixels, and a blob can be of any shape. A blob can be considered as a connected part. In a heatmap, more than one blob can be detected.

[0103] After heatmap NMS, each heatmap channel has no more than one spot. The keypoint can be the peak position of the spot's corresponding channel map. For some keypoints, their corresponding heatmap may not have the rest of the spots. In this case, the keypoint for these channels will be the peak position.

[0104] As an example, Figures 2A to 2L The figure exemplarily shows the posture estimation process performed by the human posture estimation module 130 according to some embodiments of the present disclosure, wherein: Figure 2A An object among multiple objects within a frame located by a corresponding bounding box according to some embodiments of the present disclosure is exemplarily shown. Figures 2B to 2K A partial regression heat map result of an image region defined by a bounding box for the object according to some embodiments of the present disclosure is exemplarily shown. Figure 2L The pose estimation results of the object using cross-heatmap NMS suppression according to some embodiments of the present disclosure are exemplarily shown, and the key points and the lines connecting two adjacent key points can represent the pose of the object. In some embodiments, the RGB channel image of the object can be regressed into a set of 15 heat maps, each of which can correspond to a key point. In addition, it should be noted that the number of heat maps regressed from the image of the object is not limited to this.

[0105] For example, Figure 3B The output results generated by the human body posture estimation module 130 according to some embodiments of the present disclosure are exemplarily shown. Specifically, Figure 3A The local key points of multiple objects shown in Figure 3A The region defined by the plurality of bounding boxes shown in FIG. Figure 3B As shown, each of the multiple objects corresponds to the same number (eg, 15) of predefined local key points.

[0106] Then, the estimation result of each object in each frame of the plurality of consecutive frames is converted into an openSVAI standardized data format so as to generate a standardized estimation result for each object in each frame of the plurality of consecutive frames.

[0107] When the human pose estimation module 130 is loaded into the memory 114 and executed by the processor 112 , the processor 112 may also generate a standardized estimation result for each object in each frame of a plurality of consecutive frames and then send the standardized estimation result to the human pose tracking module 140 .

[0108] In some embodiments, the human pose tracking module 140, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to track the pose of each object across multiple consecutive frames in response to receiving the standardized estimation results for each object in each frame of multiple consecutive frames. In some embodiments, the human pose tracking module 140, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to assign the same identifier to the bounding box to indicate the same object in the objects across multiple consecutive frames, and associate the key points within the bounding box with the same identifier across multiple consecutive frames, thereby establishing a pose stream for each object across multiple consecutive frames. Here, the term "pose stream" can be a set of poses indicating the same person instance (i.e., object) in multiple consecutive frames.

[0109] In some embodiments, the human pose tracking module 140 can be implemented by a flow-based pose tracker (for example, see Xiu, Y., Li, J., Wang, H., Fang, Y., Lu, C.: Pose flow: Efficient online pose tracking. In: BMVC (2018), the entire contents of which are incorporated herein by reference). Specifically, the human pose tracking module 140 can establish an association of cross-frame poses determined by the human pose estimation module 130 and thereby form a pose flow, and then reduce redundant pose flows, and use pose flow non-maximum suppression (NMS) to relink temporally non-intersecting pose flows. The details of this process can be referred to the above content.

[0110] Extensive experiments were conducted to improve the performance of the human pose tracking module 140. The performance of the pose tracker can be evaluated via MOTA, which is very strict. MOTA penalizes mismatches, false positives, and missed negatives. In order to obtain higher MOTA results, key points with low confidence scores need to be removed, thus sacrificing the recall rate of accurate key points. Table 5 illustrates the sensitivity analysis of how the key point discarding threshold affects the AP and MOTA performance (performed on the PoseTrack 2018 validation set). It can be seen from Table 5 that as the discarding threshold increases from 0.5 to 0.8, the AP of pose estimation drops from 76.3 to 69.7, while the MOTA result of pose tracking increases significantly from 40.4 to 62.4, indicating that the MOTA evaluation criterion is very sensitive to the discarding rate of key points.

[0111] Table 5

[0112]

[0113] Considering the obvious difficulty of key points (e.g., shoulders are easier to locate than ankles), the confidence distribution of each node may be uneven. Discarding key points based only on the key point confidence estimated by the posture estimator (e.g., implemented by the human posture estimation module 130) may not be an ideal strategy for posture tracking. In some embodiments, statistical information of the discard rate of key points is collected from different nodes, as shown in Table 6, which illustrates a statistical analysis of the discard rate of key points with different discard thresholds (the analysis is performed on the PoseTrack 2018 validation set, and the number indicates the percentage of key points retained after pruning). It can be seen from the respective retention rates of the key points that, from left to right, the estimation of key points is becoming more and more difficult. The simplest and most difficult nodes are shoulders and ankles, respectively. That is, the posture estimator is most confident in estimating the key points of the shoulders, while the least confident in estimating the key points of the ankles. Therefore, an adaptive key point pruner can help improve MOTA performance while maintaining a high recall rate.

[0114] Table 6

[0115]

[0116] In some embodiments, the adaptive keypoint pruner can be implemented by a convolutional neural network (CNN). In some embodiments, the adaptive keypoint pruner performs well on pruning keypoints based only on a fixed threshold, which is sensitive to the test data and the pose estimator.

[0117] Figure 1CAn exemplary block diagram schematically shows the configuration structure of the adaptive key point pruner 142 according to some embodiments of the present disclosure. In some embodiments, the adaptive key point pruner 142 may be included in the human body posture tracking module 140.

[0118] like Figure 1C As shown, the adaptive key point pruner 142 may include: a key point discarding threshold determination module 144 and a confidence score comparison module 146. In some embodiments, when the key point discarding threshold determination module 144 is loaded into the memory 114 and executed by the processor 112, it may cause the processor 112 to determine a key point discarding threshold for each predefined key point category. In some embodiments, when the confidence score comparison module 146 is loaded into the memory 114 and executed by the processor 112, it may cause the processor 112 to compare the confidence score of each key point of each object in each frame with the corresponding key point discarding threshold, discard the key points with a confidence score less than the corresponding key point discarding threshold, and retain the key points with a confidence score greater than or equal to the corresponding key point discarding threshold. Therefore, key points with lower confidence scores can be adaptively discarded based on the corresponding discarding threshold of the corresponding key point category.

[0119] After adaptive key point pruning of the key points of each object in each of the multiple consecutive frames received from the human pose estimation module 130, the retained key points of each object in each of the multiple consecutive frames can be input to the human pose tracking module 140. The pose tracking process can start from the first frame where the candidate is detected. The human pose tracking module 140 can assign an identifier to each object across multiple frames. In order to prevent the assignment of an ID to a person (i.e., an object) that has left the visible image area, the ID is only retained for a limited number of frames and then discarded. For example, Figure 3C The output results generated by the human posture tracking module 140 according to some embodiments of the present disclosure are exemplarily shown. Figure 3C As shown, Figure 3B Each of the multiple objects in the image is assigned a unique identifier, and the same object in multiple consecutive frames has the same identifier.

[0120] It should be noted that Figures 1A to 1C The computing device 100 is shown as an example. In other embodiments, the computing device 100 may include Figures 1A to 1C Fewer, different, differently arranged, or additional devices and / or modules may be depicted. Additionally or alternatively, one or more devices and / or modules of computing device 100 may perform one or more tasks described as being performed by one or more other devices and / or modules of computing device 100.

[0121] In the following, reference will be made to Figures 4 to 7 A method for tracking the posture of multiple objects according to some embodiments of the present disclosure is described in detail. It should be particularly noted that, unless otherwise specified in the present disclosure, the steps of any method can be arranged in a different order and are therefore not limited to the order shown in the accompanying drawings. In addition, the same description as the above-mentioned device for tracking the posture of multiple objects will be omitted.

[0122] Figure 4 The flowchart of the method for tracking the posture of multiple objects based on multiple single-object posture estimators according to some embodiments of the present disclosure is schematically shown. In some embodiments, the method can be Figure 1A In some embodiments, Figure 4 The method shown may correspond to part or all of the functionality of the multi-object gesture tracking application 118. The multi-object gesture tracking method may include the following steps.

[0123] In step 401, the candidate detection module 120, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to detect an object from each of a plurality of consecutive frames in a video sequence and send the detection results to the executed human pose estimation module 130. In some embodiments, the results include bounding boxes in each frame. In some embodiments, the results are described in an openSVAI format.

[0124] In step 402, the human pose estimation module 130, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to estimate the pose of each object in each of a plurality of consecutive frames using the detection results in response to receiving the detection results from the candidate detection module 120, and send the estimated pose to the executed human pose tracking module 140. In some embodiments, the estimated pose is described in an openSVAI format.

[0125] In step 403 , the human pose tracking module 140 , when loaded into the memory 114 and executed by the processor 112 , may cause the processor 112 to track the pose of each object across a plurality of consecutive frames in response to receiving the estimated pose of the object.

[0126] Figure 5 The flowchart of the method for tracking the posture of multiple objects based on multiple single-object posture estimators according to other embodiments of the present disclosure is schematically shown. In some embodiments, the method can be Figure 1A In some embodiments, Figure 5 The method shown may correspond to part or all of the functionality of the multi-object gesture tracking application 118. The multi-object gesture tracking method may include the following steps.

[0127] In step 501, when the candidate detection module 120 is loaded into the memory 114 and executed by the processor 112, the processor 112 can determine the bounding box for each object in each frame of multiple consecutive frames in the video sequence and determine the confidence score of the bounding box, and send the determined bounding box and its confidence score to the executed human posture estimation module 130.

[0128] In step 502, when the human pose estimation module 130 is loaded into the memory 114 and executed by the processor 112, the processor 112 can determine the key points of each object in each frame of the multiple consecutive frames from the area defined by the bounding box for each object in each frame of the multiple consecutive frames in response to receiving the bounding box and its confidence score, and determine the confidence score of each key point, and send the results of the key points and confidence scores to the executed human pose tracking module 140.

[0129] In some embodiments, the step of determining the key points of each object in each of the plurality of consecutive frames from the area defined by the bounding box for each object in each of the plurality of consecutive frames, and determining the confidence score of each key point may include: regressing the area defined by the bounding box for each object in each of the plurality of consecutive frames into a heat map of each object in each of the plurality of consecutive frames; suppressing the heat map of each object in each of the plurality of consecutive frames into the key points of each object in each of the plurality of consecutive frames using cross-heat map pose non-maximum suppression (NMS); and determining the confidence score of the key points of each object in each of the plurality of consecutive frames. The details of cross-heat map pose NMS will be referred to below. Figure 6 Give a description.

[0130] In some embodiments, the step of determining the key points of each object in each frame of the plurality of consecutive frames from the area defined by the bounding box for each object in each frame of the plurality of consecutive frames, and determining the confidence score of each key point may also include: applying a model integration mode combining two or more pose estimation models to determine the key points of each object in each frame of the plurality of consecutive frames, wherein the two or more pose estimation models include at least a first model and a second model. In some embodiments, the model integration mode may include an average mode, in which, for all key points of each object in each frame of the plurality of consecutive frames, a weight of 0.5 is assigned to the pose estimation result for the first model, and a weight of 0.5 is assigned to the pose estimation result for the second model. In other embodiments, the model integration mode includes an expert mode, in which, for each key point of each object in each frame of the plurality of consecutive frames, a weight of 1 is assigned to the pose estimation result with a higher average accuracy AP in the pose estimation result for the first model and the pose estimation result for the second model, and a weight of 0 is assigned to another pose estimation result in the pose estimation result for the first model and the pose estimation result for the second model. In some embodiments, one of the first model and the second model is epoch 291, and the other of the first model and the second model is epoch 293.

[0131] In step 503, when the human posture tracking module 140 is loaded into the memory 114 and executed by the processor 112, the processor 112 can use the adaptive key point pruning device to perform adaptive key point pruning on the key points of each object in each frame of the plurality of consecutive frames in response to receiving the posture estimation result from the posture estimation module 130. The details of the execution process of the adaptive key point pruning device 142 will be referred to below. Figure 7 Give a description.

[0132] In step 504, the human pose tracking module 140, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to assign the same identifier to the bounding box after performing adaptive keypoint pruning to indicate the same object in the objects across multiple consecutive frames.

[0133] In step 505, when the human posture tracking module 140 is loaded into the memory 114 and executed by the processor 112, the processor 112 can associate the key points within the bounding boxes with the same identifier across multiple consecutive frames after assigning the same identifier to the bounding box indicating the same object in the objects across multiple consecutive frames to establish a posture stream for each object across multiple consecutive frames.

[0134] Figure 6A flowchart of a method for suppressing a heat map of an object as a key point using a cross heat map NMS according to some embodiments of the present disclosure is exemplarily shown. In some embodiments, the method may be Figure 1B The cross-heatmap gesture NMS module 132 is shown to implement the cross-heatmap gesture NMS suppression method. The cross-heatmap NMS suppression method may include the following steps.

[0135] In step 601, when the heat map acquisition module 133 is loaded into the memory 114 and executed by the processor 112, the processor 112 can receive a set (e.g., N) of heat maps regressed from an image of an object, where N is a positive integer, and send the set of heat maps to the executed spot collection module 134.

[0136] In step 602, the speckle collection module 134, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to collect speckles from all N heat maps for suppression in response to receiving the set of heat maps, and send the collected speckles to the executed speckle suppression module 135.

[0137] In step 603, when the speckle suppression module 135 is loaded into the memory 114 and executed by the processor 112, the processor 112 can determine the spot with the maximum response in response to receiving the collected spots, suppress other spots in the same heat map and spots in other heat maps that are very close to the determined spot in the image coordinate system, and delete the heat map from the group of heat maps (N=N-1).

[0138] In step 604, the heat map quantity comparison module 136, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to compare the current heat map quantity N=N-1 with 0. If N=N-1 is greater than zero, the process returns to step 603, and if N=N-1 is not greater than zero, the process terminates. Here, it should be noted that the term "termination" refers to the termination of the suppression process for the object, while other objects can follow the same suppression process. Using the above-mentioned cross-heat map NMS method, the key points of each object in each frame of multiple consecutive frames can be determined.

[0139] Figure 7 A flowchart of a method for pruning key points using an adaptive key point pruner according to some embodiments of the present disclosure is exemplarily shown. In some embodiments, the method may be Figure 1C The adaptive keypoint pruner 142 is shown to implement the method of pruning keypoints.

[0140] In step 701 , when the key point discarding threshold determination module 144 is loaded into the memory 114 and executed by the processor 112 , the processor 112 may determine a key point discarding threshold for each predefined key point category and send the key point discarding threshold to the confidence score comparison module 146 .

[0141] In step 702, the confidence score comparison module 146, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to determine whether the confidence score of each key point of each object in each frame is greater than or equal to the corresponding key point discarding threshold in response to receiving the key point discarding threshold. If yes, proceed to step 703, and if not, proceed to step 704.

[0142] In step 703, the confidence score comparison module 146, when loaded into the memory 114 and executed by the processor 112, may cause the processor 112 to retain the keypoint in response to the confidence score of the keypoint being greater than or equal to the corresponding keypoint discard threshold.

[0143] In step 704, the confidence score comparison module 146, when loaded into the memory 114 and executed by the processor 112, can cause the processor 112 to discard the key point in response to the confidence score of the key point being less than the corresponding key point discard threshold. Using the above key point pruning method, key points with lower confidence scores can be adaptively discarded based on the corresponding discard threshold of the corresponding key point category.

[0144] As an example, the final performance of a partial test set of PoseTrack2018 implemented by the above apparatus and method is shown in Table 7, which shows the single-frame pose estimation results on the partial test set of PoseTrack 2018, and Table 8 shows the multi-frame pose tracking results on the partial test set of PoseTrack 2018.

[0145] Table 7

[0146]

[0147] Table 8

[0148]

[0149] In summary, some embodiments of the present disclosure provide an accurate and robust multi-object pose tracking system, the system comprising a state-of-the-art object detector, a single person pose estimator, and a human pose tracker. In some embodiments, the multi-object pose tracking system is configured to select a general object detector in the state of the art to detect candidates, estimate the corresponding human pose using a cascaded pyramid network, and render key point associations across frames using a flow-based pose tracker, i.e., assign a unique and temporally consistent ID to each candidate for multi-target pose tracking. Extensive ablation experiments were conducted to validate various choices and configuration structures of the model and obtain better performance.

[0150] It should be noted that the submodules of the multi-object posture tracking system based on multiple single-object posture estimators may be universal, for example, the human posture estimation module 130 and the human posture tracking module 140 may be universal, and these submodules are replaceable and upgradeable.

[0151] Advantages of the present disclosure include, among others: (1) providing an end-to-end human pose estimation and tracking system, the system comprising an object detection module for detecting an object in a series of frames, a single person pose estimation module for estimating the pose of an object in a series of frames, and a pose tracking module for tracking the pose of an object in a series of frames based on the estimated pose; (2) the pose estimation module combines the advantages of different models, wherein the weight of each key point may be different when different models are used; and (3) providing an openSVAI standardized data format for the entire top-down approach.

[0152] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0153] The embodiments are selected and described in order to explain the principles of the present disclosure and its practical application, so as to enable other persons skilled in the art to utilize the present disclosure and various embodiments and make various modifications to suit the intended specific use. Without departing from the spirit and scope of the present application, alternative embodiments are apparent to those skilled in the art to which the present disclosure belongs. Therefore, the scope of the present disclosure is limited by the appended claims rather than the foregoing description and the exemplary embodiments described therein.

[0154] References

[0155] 1.Andriluka,M.,Iqbal,U.,Milan,A.,Insafutdinov,E.,Pishchulin,L.,Gall,J.,Schiele,B.,Posetrack:A benchmark for human pose estimation andtracking.In:Proceedings of the IEEE Conference on Computer Vision and PatternRecognition,2018,pp.5167-5176.

[0156] 2.Cao,Z.,Simon,T.,Wei,S.E.,Sheikh,Y.,Realtime multi-object 2d poseestimation using part affinity fields,CVPR,2017,pp.7291-7299.

[0157] 3.Chen,Y.,Wang,Z.,Peng,Y.,Zhang,Z.,Yu,G.,Sun,J.,Cascaded PyramidNetwork for Multi-object Pose Estimation,CVPR,2018,pp.7103-7112.

[0158] 4.Dai,J.,Li,Y.,He,K.,Sun,J.:R-FCN:Object detection via region-basedfully convolutional networks,Advances in neural information processingsystems,2016.

[0159] 5.Dai,J.,Qi,H.,Xiong,Y.,Li,Y.,Zhang,G.,Hu,H.,Wei,Y.,Deformableconvolutional networks,2017,arXiv:1703.06211.

[0160] 6.Girdhar,R.,Gkioxari,G.,Torresani,L.,Paluri,M.,Tran,D.,Detect-and-track:Efficient pose estimation in videos.In:Proceedings of the IEEEConference on Computer Vision and Pattern Recognition,2018,pp.350-359.

[0161] 7.Girshick,R.:Fast R-CNN,Proceedings of the IEEE internationalconference on computer vision,2015,pp.1440-1448.

[0162] 8.Hao-Shu Fang,Shuqin Xie,Y.W.T.,Lu,C.,RMPE:Regional multi-objectpose estimation,ICCV,2017,arXiv:1612.00137.

[0163] 9.He,K.,Gkioxari,G.,Dollar,P.,Girshick,R.:Mask R-CNN,2017IEEEInternational Conference on Computer Vision(ICCV),2017,pp.2961-2969.

[0164] 10.He,K.,Zhang,X.,Ren,S.,Sun,J.,Deep residual learning forimagerecognition,CVPR,2016,pp.770-778.

[0165] 11.Insafutdinov,E.,Andriluka,M.,Pishchulin,L.,Tang,S.,Levinkov,E.,Andres,B.,Schiele,B.,Arttrack:Articulated multi-object tracking inthe wild,CVPR,2017,pp.6457-6465.

[0166] 12.Iqbal,U.,Milan,A.,Gall,J.:Posetrack,Joint multi-objectposeestimation and tracking,IEEE Conference on Computer Vision andPatternRecognition(CVPR),2017,pp.2011-2020.

[0167] 13.Lin,T.Y.,Dollar,P.,Girshick,R.B.,He,K.,Hariharan,B.,Belongie,S.J.,Feature pyramid networks for object detection,CVPR,2017,pp.2117-2125.

[0168] 14.Newell,A.,Huang,Z.,Deng,J.,Associative embedding:End-to-endlearning for joint detection and grouping,Advances in NeuralInformationProcessing Systems,2017,pp.1-11.

[0169] 15.Ning,G.,Zhang,Z.,He,Z.,Knowledge-guided deep fractalneuralnetworks for human pose estimation,IEEE Transactions onMultimedia,2018,arXiv:1705.02407.

[0170] 16.Papandreou,G.,Zhu,T.,Kanazawa,N.,Toshev,A.,Tompson,J.,Bregler,C.,Murphy,K.,Towards accurate multi-object pose estimation inthe wild,CVPR,2017,pp.4903-4911.

[0171] 17.Shao,S.,Zhao,Z.,Li,B.,Xiao,T.,Yu,G.,Zhang,X.,Sun,J.,Crowdhuman:Abenchmark for detecting human in a crowd,2018,arXiv:1805.00123.

[0172] 18.Xia,F.,Wang,P.,Chen,X.,Yuille,A.L.,Joint multi-objectposeestimation and semantic part segmentation,CVPR,2017,pp.6769-6778.

[0173] 19.Xiao,B.,Wu,H.,Wei,Y.,Simple baselines for human poseestimation andtracking,ECCV,2018,pp.1-16.

[0174] 20.Xiu,Y.,Li,J.,Wang,H.,Fang,Y.,Lu,C.:Pose flow,Efficientonline posetracking,BMVC,2018,arXiv:1802.00977.

Claims

1. A posture tracking method for multiple objects, comprising: detecting, by one or more processors, objects based on a bounding box of each object determined from each of a plurality of consecutive frames in a video sequence; estimating, by the one or more processors, a pose of each object in each of the plurality of consecutive frames based on key points of the object in each of the bounding boxes determined based on a model integration mode; as well as tracking, by the one or more processors, a pose of each of the objects across the plurality of consecutive frames; The step of detecting an object based on a bounding box of each object determined from each of a plurality of consecutive frames in a video sequence comprises: determining, by the one or more processors, a bounding box for each of the objects from each of the plurality of consecutive frames; and determining, by the one or more processors, a confidence score for the bounding box for the object in one of the plurality of consecutive frames; The step of estimating the posture of each object in each frame of the plurality of consecutive frames comprises: determining, by the one or more processors, key points of the object in one of the plurality of consecutive frames from a region defined by the bounding box for the object in one of the plurality of consecutive frames; and determining, by the one or more processors, a confidence score for the key point of the object in one of the plurality of consecutive frames; The step of determining the key points of each object in the bounding box based on the model integration mode includes: A model integration mode combining two or more pose estimation models is applied by the one or more processors to determine key points of the object in one of the plurality of consecutive frames.

2. The method according to claim 1, wherein: The two or more posture estimation models include at least a first model and a second model, and the model integration mode includes an expert mode. In the expert mode, for each key point of each object in each frame of the multiple consecutive frames, a weight of 1 is assigned to a posture estimation result with a higher average accuracy AP among the posture estimation results for the first model and the posture estimation results for the second model, and a weight of 0 is assigned to another posture estimation result among the posture estimation results for the first model and the posture estimation results for the second model.

3. The method according to claim 1, wherein: The two or more posture estimation models include at least a first model and a second model, and the model integration mode includes an average mode, in which, for all key points of each object in each frame of the multiple consecutive frames, a weight of 0.5 is assigned to the posture estimation result for the first model, and a weight of 0.5 is assigned to the posture estimation result for the second model.

4. The method according to claim 1, wherein: The two or more pose estimation models include at least a first model and a second model, one of the first model and the second model is epoch 291, and the other of the first model and the second model is epoch 293.

5. The method according to claim 1, wherein: The step of tracking the pose of each object across the plurality of consecutive frames comprises: assigning, by the one or more processors, a same identifier to the bounding box to indicate a same object among the objects across the plurality of consecutive frames; and The one or more processors are configured to associate key points within the bounding box having the same identifier across the plurality of consecutive frames to establish a pose stream for each object across the plurality of consecutive frames.

6. The method according to claim 1, wherein: The step of tracking the pose of each object across the plurality of consecutive frames further comprises: The one or more processors adaptively prune key points of each object in each frame of the plurality of consecutive frames using an adaptive key point pruner.

7. The method according to claim 6, wherein: The step of using an adaptive key point pruner to adaptively prune the key points of each object in each frame of the plurality of consecutive frames comprises: determining, by the one or more processors, a keypoint discarding threshold for each predefined keypoint category, wherein each keypoint of each object in each frame of the plurality of consecutive frames belongs to one of the predefined keypoint categories; discarding, by the one or more processors, a keypoint, wherein the discarded keypoint has a confidence score less than a keypoint discard threshold for a predefined keypoint category to which the discarded keypoint belongs; and Keypoints are retained, by the one or more processors, wherein the retained keypoints have a confidence score greater than or equal to a keypoint discarding threshold for a predefined keypoint category to which the retained keypoint belongs.

8. The method according to claim 1, wherein: The step of detecting an object from each of a plurality of consecutive frames in a video sequence is implemented by a deformable feature pyramid network (FPN), wherein the deformable feature pyramid network is determined based on an object detector selection mechanism.

9. The method according to claim 1, wherein: The step of detecting an object from each of a plurality of consecutive frames in the video sequence further comprises: The one or more processors convert the detection result of each object in each frame of the plurality of consecutive frames into an openSVAI standardized data format to generate a standardized detection result.

10. The method according to claim 1, wherein: The step of estimating the pose of each object in each of the plurality of consecutive frames further comprises: The one or more processors convert the estimation result for each object in each frame of the plurality of consecutive frames into an openSVAI standardized data format to generate a standardized estimation result.

11. A method for tracking the postures of multiple objects, comprising: The step of detecting, by one or more processors, objects based on a bounding box of each object determined from each of a plurality of consecutive frames in a video sequence comprises: determining, by the one or more processors, a bounding box for each of the objects from each of the plurality of consecutive frames; determining, by the one or more processors, a confidence score for the bounding box for the object in one of the plurality of consecutive frames; estimating, by the one or more processors, a pose of each object in each of the plurality of consecutive frames based on key points of the object in each of the bounding boxes determined based on a model integration mode; tracking, by the one or more processors, a pose of each of the objects across the plurality of consecutive frames; The step of estimating the posture of each object in each frame of the plurality of consecutive frames comprises: regressing, by the one or more processors, a region defined by the bounding box for the object in one of the plurality of consecutive frames into a heat map of the object in one of the plurality of consecutive frames; Suppressing, by the one or more processors, the heatmap of the object in one of the plurality of consecutive frames to key points of the object in one of the plurality of consecutive frames using cross-heatmap pose non-maximum suppression (NMS); and A confidence score for a key point of the object in one of the plurality of consecutive frames is determined by the one or more processors.

12. A posture tracking device for multiple objects, comprising: processor; as well as A memory having instructions stored thereon, which, when executed by the processor, cause the processor to: detecting objects based on a bounding box of each object determined from each of a plurality of consecutive frames in a video sequence; estimating a pose of each object in each of the plurality of consecutive frames based on key points of the object in each of the bounding boxes determined based on a model integration mode; as well as tracking a pose of each of the objects across the plurality of consecutive frames; Wherein, when the instruction is executed by the processor, the processor is also caused to: determining a bounding box for each of the objects from each of the plurality of consecutive frames; and determining a confidence score for the bounding box for the object in one of the plurality of consecutive frames; Wherein when the instruction is executed by the processor, the processor is further caused to: Determining key points of the object in one of the plurality of consecutive frames from an area defined by the bounding box for the object in one of the plurality of consecutive frames; and determining a confidence score for the key point of the object in one of the plurality of consecutive frames; Wherein when the instruction is executed by the processor, the processor is further caused to: A model integration mode is used to combine two or more pose estimation models to determine key points of the object in one of the plurality of consecutive frames.

13. The apparatus according to claim 12, wherein when the instructions are executed by the processor, the processor is further caused to: assigning a same identifier to the bounding box to indicate a same one of the objects across the plurality of consecutive frames; and Key points within the bounding box having the same identifier across the plurality of consecutive frames are associated to establish a pose flow for each object across the plurality of consecutive frames.

14. The apparatus according to claim 12, wherein when the instructions are executed by the processor, the processor is further caused to: converting the detection result of each object in each frame of the plurality of consecutive frames into an openSVAI standardized data format to generate a standardized detection result; and The estimation result for each object in each frame of the plurality of consecutive frames is converted into an openSVAI standardized data format to generate a standardized estimation result.

15. A non-transitory computer-readable medium having computer-executable instructions stored thereon, which, when executed by a processor of a computing device, cause the processor to: detecting objects based on a bounding box of each object determined from each of a plurality of consecutive frames in a video sequence; estimating a pose of each object in each of the plurality of consecutive frames based on key points of the object in each of the bounding boxes determined based on a model integration mode; as well as tracking a pose of each of the objects across the plurality of consecutive frames; in, The step of detecting objects based on a bounding box of each object determined from each of a plurality of consecutive frames in a video sequence comprises: determining a bounding box for each of the objects from each of the plurality of consecutive frames; and determining a confidence score for the bounding box for the object in one of the plurality of consecutive frames; The step of estimating the posture of each object in each frame of the plurality of consecutive frames comprises: Determining key points of the object in one of the plurality of consecutive frames from an area defined by the bounding box for the object in one of the plurality of consecutive frames; and determining a confidence score for the key point of the object in one of the plurality of consecutive frames; The step of determining the key points of each object in the bounding box based on the model integration mode includes: A model integration mode is used to combine two or more pose estimation models to determine key points of the object in one of the plurality of consecutive frames.

Images