Object detection methods, systems, and computer program products

Abstract

Techniques for performing object tracking of objects in video images using both an object detection algorithm and an object tracking algorithm. Because the object detection algorithm is generally slower but more accurate than the object tracking algorithm, only certain sampled video frames are subjected to the object detection algorithm. Non-sampled frames use the object tracking algorithm to locate objects, using the most recent previous object detection frame as a starting frame. In this way, the object tracking algorithm is periodically, effectively calibrated using the object detection algorithm, by hiding the latency introduced by the object detection algorithm using the lagging detection results.

Description

BACKGROUND

[0001] The present invention relates generally to the field of machine logic for object tracking and machine logic for object detection.

[0002] It is known that computer systems can perform "object detection" (referred to herein as object detection systems). Object detection systems typically include a graphics processing unit (GPU). Object detection is a computer technology involving computer vision and image processing that deals with detecting the presence of "semantic objects" of some predefined class (such as humans, buildings or cars) in digital images (e.g. videos). Some domains of object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance. It is used for computer vision tasks such as face detection, face recognition, video object co-segmentation. It is also used in conjunction with "object tracking" (discussed in the following paragraph). Each object class has its own special features that help to classify that class. For example, all circles are circular. Object class detection uses these special features. Conventional object detection typically uses either a machine learning based approach or a deep learning based approach. For machine learning type object detection, it is necessary to first define the features using one of the methods listed below, and then use a technique such as a support vector machine (SVM) for classification. Deep learning type object detection, which is capable of end-to-end object detection without the need to specifically define features, and is typically based on a convolutional neural network (CNN). The different types of machine learning methods mentioned above include: (i) Viola-Jones object detection framework based on Haar features; (ii) scale-invariant feature transform (SIFT); and / or (iii) histogram of oriented gradients (HOG) features. Different types of deep learning methods include: (i) region proposal (R-CNN, Fast R-CNN, Faster R-CNN); (ii) single shot multibox detector; and (iii) you only look once (YOLO).

[0003] It is well known that computer systems can perform "object tracking" (referred to herein as an object tracking system) through object tracker software running on a central processing unit (CPU). Object tracking is also sometimes referred to as "video tracking". As of August 14, 2019, the Wikipedia entry for the term video tracking states as follows: "Video tracking is the process of locating a moving object (or multiple objects) over time using a camera. It has multiple uses, some of which are: human-computer interaction, security and surveillance, video communication and compression, augmented reality, traffic control, medical imaging, and video editing. Due to the amount of data contained in a video, video tracking can be a time-consuming process. Further adding complexity is the possibility that object recognition techniques can need to be used for tracking, which is itself a challenging problem. An example of visual servoing in which a robotic hand grasps a ball through object tracking with visual feedback processed by a high-speed image processing system. The purpose of video tracking is to associate a target object in successive video frames. Association can be especially difficult when the object moves quickly relative to the frame rate. Another scenario that adds complexity to the problem is when the object being tracked changes orientation over time. For these cases, video tracking systems typically employ a motion model that describes how the image of the target can change for different possible motions of the object.

[0004] An article entitled "Stable tracking extension keyframe detection for multiple 3D object tracking" appeared in IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 17, NO. 11, NOVEMBER 2011 ("Park et al. article") discloses as follows: "We now explain our stable tracking algorithm, which combines object detection and frame-by-frame tracking. Frame-by-frame tracking has two purposes: it completes the detection of an object whenever it exists, and it stabilizes the estimated pose, thus removing the jittering effect that individual detection would produce. We extract feature points in each input frame and match them with the feature points extracted in the previous frame using standard techniques based on cross-correlation and local search. In comparison with KLT trackers, it allows us to easily handle appearing and disappearing feature points, and prevents drift in the tracked position. We call these feature points 'temporal key points'. SUMMARY

[0005] According to aspects of the present application, there is provided a method, computer program product, and / or system that performs the following operations (not necessarily in the following order): (i) receiving a sequential and time-sequential plurality of video frames from a video image comprising a moving object; (ii) sampling a plurality of selected video frames from the plurality of video frames such that one (1) out of every N frames is a selected video frame, where N is an integer greater than one (1); (iii) performing a convolutional neural network (CNN) object detection algorithm on the plurality of selected video frames to locate the moving object in each of the selected video frames and obtain a respective object location for each of the selected video frames to obtain sampled frame object locations for the moving object; and (iv) performing object tracking on the plurality of video frames to locate the moving object in each of the video frames to obtain object tracking locations for the moving object, wherein the object tracking comprises: (a) for each given selected video frame of the plurality of video frames, using the sampled frame object location as the object tracking location for the given selected video frame, (b) for each given video frame that is not a selected video frame of the plurality of video frames, applying an object tracking algorithm in the object tracking algorithm using a most recent previous sampled frame object location as a starting frame to obtain the object tracking location for the given video frame that is not a selected frame, such that the sampled frame object locations are used to calibrate the object tracking locations for video frames that are not selected, and (c) concealing a detection latency associated with the performance of the object detection algorithm on the plurality of selected video frames.

[0006] According to aspects of the present invention, there is provided a method, computer program product, and / or system that performs the following operations (not necessarily in the following order): (i) receiving a sequential and time-sequential plurality of video frames from a video image comprising a moving object; (ii) sampling a plurality of selected video frames from the plurality of video frames such that one (1) out of every N frames is a selected video frame, where N is an integer greater than one (1); (iii) performing an object detection algorithm on the plurality of selected video frames to locate the moving object in each of the selected video frames and obtain a respective object location for each of the selected video frames from the plurality of video frames to obtain sampled frame object locations of the moving object; and (iv) performing a Kernelized Correlation Filter (KCF) object tracking on the plurality of video frames to locate the moving object in each of the video frames to obtain object tracking locations of the moving object, wherein the object tracking comprises: (a) for each given selected video frame from the plurality of video frames, using the sampled frame object location as the object tracking location for the given selected video frame, (b) for each given video frame that is not a selected video frame from the plurality of video frames, applying an object tracking algorithm in the object tracking algorithm using the most recent previously sampled frame object location as a starting frame to obtain the object tracking location for the given video frame that is not a selected frame, such that the sampled frame object locations are used to calibrate the object tracking locations for video frames that are not selected video frames, and (c) concealing a detection latency associated with the performance of the object detection algorithm on the plurality of selected video frames. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a block diagram of a first embodiment of a system according to the present invention;

[0008] Figure 2A is a flowchart illustrating a first embodiment method performed at least in part by the first embodiment system;

[0009] Figure 2B is a supplemental flowchart to the flowchart of Figure 2A illustrating operations and timing of operations at a more granular level;

[0010] Figure 3 is a block diagram illustrating a machine logic (e.g., software) portion of the first embodiment system;

[0011] Figures 4A-4J is a video frame and associated video overlay generated by the first embodiment system;

[0012] Figure 5 is a block diagram of a second embodiment of a system according to the present invention;

[0013] Figure 6 is a table that is helpful in understanding various embodiments of the present invention;

[0014] Figure 7 is a block diagram illustrating a data flow according to an embodiment of the present invention; and

[0015] Figure 8 is a timeline view of a third embodiment of the method according to the present invention. DETAILED DESCRIPTION

[0016] Some embodiments of the present invention relate to performing object tracking of objects in video images using both an object detection algorithm and an object tracking algorithm. Because the object detection algorithm is generally slower but more accurate than the object tracking algorithm, only particular, sampled video frames are subjected to the object detection algorithm. Non-sampled frames use the object tracking algorithm to locate objects, using the most recent previous object detection frame as a starting frame. In this way, the object tracking algorithm is periodically, effectively calibrated using the object detection algorithm, while using the lagging detection results to hide the latency introduced by the object detection algorithm. This detailed description section is divided into the following subsections: (i) hardware and software environment; (ii) example embodiments; (iii) further comments and / or embodiments; and (iv) definitions.

[0017] I. Hardware and Software Environment

[0018] The present invention can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0019] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0020] Here, "storage device" is defined as any which is manufactured or adapted for the purpose of storing computer code and which is in use by a computer processor. Storage devices are typically (but not exclusively) volatile memory devices such as DRAM, SRAM, and Flash. A single "storage device" can have multiple discrete portions (e.g., a set of six solid state storage devices each located in a separate laptop computer collectively storing a single computer program) and / or can use multiple storage media (e.g., a set of computer code stored partially as magnetic domains in a computer's non-volatile memory and partially in a set of semiconductor switches in the computer's volatile memory). The term "storage media" is taken to include instances in which multiple different types of storage media are used in conjunction with each other (e.g., a combination of disk and flash memory that operates as a single storage device).

[0021] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0022] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0023] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0024] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0025] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0026] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0027] like Figure 1 As shown, the networked computer system 100 is an embodiment of the hardware and software environment used in various embodiments of the present invention. The networked computer system 100 includes: a server subsystem 102 (sometimes more simply referred to herein as subsystem 102); client subsystems 104, 106, 108, 110, 112; and a communication network 114. The server subsystem 102 includes: a server computer 200; a communication unit 202; a processor set 204; an input / output (I / O) interface set 206; a memory 208; permanent memory 210; a display 212; external devices 214; a camera 215; random access memory (RAM) 230; a cache 232; and a program 300.

[0028] Subsystem 102 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, personal digital assistant (PDA), smartphone, or any other type of computer (see the definition of "computer" in the Definitions section below). Program 300 is a collection of machine-readable instructions and / or data for creating, managing, and controlling certain software functions, which will be described in detail below in the Example Embodiments section of the Detailed Implementation section.

[0029] Subsystem 102 is capable of communicating with other computer subsystems via communication network 114. Network 114 may be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of both, and may include wired, wireless, or fiber optic connections. Generally, network 114 may be any combination of connections and protocols that support communication between server and client subsystems.

[0030] The subsystem 102 is shown as a block diagram with many double-headed arrows. These double-headed arrows (without separate reference numbers) represent a communication fabric that provides communication between the various components of the subsystem 102. This communication fabric can be implemented with any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within a computer system. For example, the communication fabric can be implemented at least in part with one or more buses.

[0031] The memory 208 and the persistent storage 210 are computer readable storage media. In general, the memory 208 can include any suitable volatile or non-volatile computer readable storage media. It is further noted that now and / or in the near future: (i) the external device 214 can be capable of providing some or all of the memory for the subsystem 102; and / or (ii) a device external to the subsystem 102 can be capable of providing memory for the subsystem 102. The memory 208 and the persistent storage 210: (i) store data of shorter duration than the data stored in the signals in transit; and (ii) store data for the longer term such as for archiving. In this embodiment, the memory 208 is a volatile memory and the persistent storage 210 provides non-volatile storage. The media used by the persistent storage 210 can also be removable. For example, a removable hard drive can be used for the persistent storage 210. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer readable storage medium, also a part of the persistent storage 210.

[0032] The communication unit 202 provides communication with other data processing systems or devices external to the subsystem 102. In these examples, the communication unit 202 includes one or more network interface cards. The communication unit 202 can provide communication over either or both of physical and wireless communication links. Any of the software modules discussed herein can be downloaded to the persistent storage device (such as the persistent storage 210) through the communication unit (such as the communication unit 202).

[0033] The I / O interface set 206 allows for data input and output to other devices that can be connected in data communication with the server computer 200 locally. For example, the I / O interface set 206 provides a connection to the external device set 214. The external device set 214 generally includes devices such as a keyboard, keypad, touch screen, and / or some other suitable input device. The external device set 214 can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present application, e.g., the program 300, can be stored on such portable computer-readable storage media. The I / O interface set 206 also connects to a display 212 data communicationally. The display 212 is a display device that provides a mechanism to show data to a user, and can be, for example, a computer monitor or a smartphone display screen.

[0034] In this embodiment, the program 300 is stored in the persistent storage 210 for access and / or execution by one or more computer processors of the processor set 204, typically through one or more memories of the memory 208. Those skilled in the art will appreciate that the program 300 can be stored in a more highly distributed manner during its runtime and / or when it is not running. The program 300 can include machine-readable and executable instructions and / or substantive data (i.e., the type of data stored in a database). In this particular embodiment, the persistent storage 210 includes a magnetic hard disk drive. To name a few possible variations, the persistent storage 210 can include a solid state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage media that is capable of storing program instructions or digital information.

[0035] The programs described herein are identified based upon the application that implements the programs in a particular embodiment of the application. However, it should be appreciated that any particular program term(s) are used herein for convenience, and thus the application should not be limited to use solely in any specific application identified and / or implied by such term(s).

[0036] The description of various implementations of the application have been presented for purposes of illustration but is not intended to be exhaustive or limited to the implementations disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0037] II. EXEMPLARY EMBODIMENTS

[0038] AsFigure 1 As shown, the networked computer system 100 is an environment in which the example methods according to the present invention can be executed. Figure 2A As shown, flowchart 250 illustrates an example method according to the present invention. Figure 3 As shown, program 300 executes or controls the execution of at least some of the method operations in flowchart 250. This will now be extensively referred to in the following paragraphs. Figure 1 , 2A The following section, box 3, will be used to discuss this approach and the associated software.

[0039] Processing begins in operation S255, where camera 215 of external device group 214 continuously feeds video frames to receive frame modules (“mod”) 302. Ten of these frames are respectively in… Figure 4A Video frames 400a, b, c, d, e, f, g, h, i, j are shown, labeled B, C, D, E, F, G, H, I, J. In this example, video frames are captured at a rate of 60 frames per second. This example will focus on ten (10) video frames 400a, b, c, d, e, f, g, h, i, j.

[0040] The processing proceeds to operation S260, in which the first video frame in each sequence set of ten (10) video frames is sampled and an object detection algorithm 304 is performed on it. Alternatively, other sampling rates may be used to determine the frequency at which frames are sampled (or selected) to apply the object detection algorithm 304. In this example, the first of the set of ten (10) sequence frames is Figure 4A Frame 400a. In this example, the detected object is a flying / bouncing ball appearing as ball 404a in frame 400a. By detecting ball 404a, object detection algorithm 304 detects the position of ball 404a, as shown by the superimposed arrow 402a in frame 400a.

[0041] Processing continues to operation S265, in which the next nine (9) video frames 400b, c, d, e, f, g, h, i, j have an object tracking algorithm 306 applied to them. During the application of the object tracking algorithm, the ball position determined by the object detection algorithm is used as the starting position of the object tracking algorithm. In this way, in operation S260, the starting position is periodically calibrated every 1 / 6 second using the result of the object detection algorithm. The positions of the balls 404b, c, d, e, f, g, h, i, j in these subsequent nine (9) frames are respectively determined by... Figure 4B The superimposed arrows 402b, c, d, e, f, g, h, i, j indicate the numbers C, D, E, F, G, H, I, and J.

[0042] Processing proceeds to operation S270, where the output 308 outputs ten (10) frames 400a,b,c,d,e,f,g,h,i,j of the video signal 302 with the superimposed arrows 402a,b,c,d,e,f,g,h,i,j superimposed thereon, the superimposed arrows 402a,b,c,d,e,f,g,h,i,j indicating the determined position of the ball in each respective frame by the algorithms 304 and 306, if applicable. The ten (10) frames are output to conceal the detection delay imparted by use of the object detection algorithm that is slower relative to the object tracking algorithm.

[0043] Processing proceeds to operation S275, where the concealment delay module 310 conceals the detection delay imparted by use of the object detection algorithm that is slower relative to the object tracking algorithm. More specifically, the module 310 conceals the detection delay by using the lagging detection results. The module 310 outputs the ten (10) frames 400a,b,c,d,e,f,g,h,i,j in sequence order to the client subsystems 104, 106, 108, 110, and 112 over the communication network 114. As shown in the flowchart 250, after operation S275, processing loops back to operation S255 for processing the next ten (10) video frames.

[0044] Note that the foregoing discussion discusses the method of the flowchart 250 at a relatively general level, and can be somewhat misleading as to the detection delay that is concealed at a more granular level that is not further explained. This further explanation is now provided with reference to the flowchart 299 of Figure 2B , which includes the following operations (the process flow among and between these operations is shown in Figure 2B ): S280; S281; S282; S283; S284; S285; S286; S287; S288; S289; and S290.

[0045] As can be seen in the flowchart 299, in this example, only every eleventh (11 th) frame object detection. When object detection (a relatively time-consuming process) is performed on a frame (i.e., frame N in the example of flowchart 299), object tracking is performed on eleven frames (i.e., frame N and frames N+1 through N+10). This is shown at operations S280, S282, S283, and S285 of flowchart 299. After object detection is completed (see operation S285), the object tracker is calibrated using the object detection results (i.e., in this example, the object detection results for frame N). This calibration is shown at operations S286 and S287. By performing object detection and associated tracker calibration intermittently (e.g., periodically, every X frames) and while continuously performing object tracking on every frame, detection latency is hidden. While this example performs object detection every eleven frames, other periods can be used. Alternatively, object detection can be performed intermittently at variable increments of frames (e.g., embodiments in which the frequency of object detection depends on the number of tracked objects or the complexity of the video image). Also, while the example of flowchart 299 uses frames N+3, N+8, and N+10 in calibration, different amounts of non-object detection frames can alternatively be used in the calibration operations.

[0046] III. Other Comments and / or Implementations

[0047] Some embodiments of the present invention recognize the following facts, potential problems, and / or potential areas of improvement relative to the prior art: (i) object detection and object tracking are different things; (ii) object detection is typically characterized by: (a) too slow to handle live video, (b) faster-rcnn object detection on PowerAI Vision can only achieve ~13 fps (frames per second) even on a powerful P100 GPU (graphics processing unit), (b) difficult to handle live video with 25 fps,

[0048] (c) very expensive cost (2 P100 GPUs can support 1 live stream), and (d) delayed output due to detection latency; (iii) traditional object tracking is typically characterized by: (a) unsupervised and automatic identification with a single starting frame and bounding box, (b) fast (in other words, KCF (Kernelized Correlation Filter) can achieve ~200 FPS on a CPU (central processing unit)), and (c) long-term error propagation; and / or (iv) existing detection and tracking fusion works are by detection tracking, i.e., by matching detected objects (e.g., bbox, features, etc.) to track.

[0049] As Figure 5As shown, frame flowchart 500 illustrates an embodiment of the present invention. This embodiment includes the following features, characteristics, operations, and / or advantages: (i) object tracking is performed via latency detection; (ii) the object detector detects sampled frames and its output is used to calibrate the object tracker; and (iii) the object tracker directly processes consecutive frames and outputs; and (iv) improved video processing throughput, reduced latency, and reduced GPU workload. Figure 6 As shown in the table, Table 600 illustrates features associated with detection and tracking data for frame and object identification. Figure 7 As shown, the tracker update workflow diagram 700 includes the following items: detected bboxes block 702; cached tracked bboxes block 704; bbox mapping block 706; cached tracker bboxes and status block 708; new bboxes from detector block 710; isolated tracker bboxes block 712; mapped bboxes block 714; create new tracker block 716; keep active / delete trackers block 718; tracker parameter calibration block 720; API output block 722; and tracker block 724.

[0050] like Figure 8 As shown, the tracker parameter calibration graph 800 has the following characteristics: (i) both the detected frames and keyframes are used to calibrate the tracker parameters; (ii) keyframes are frames with relatively large changes between two frames; (iii) parameter calibration includes the detected frames and uses the detected ROIs (regions of interest, sometimes also called bounding boxes or bboxes) to directly train the tracker parameters, while ignoring the tracker's bbox output; and (iv) parameter calibration includes keyframes: self-training with both parameter updates and bbox output updates.

[0051] Some embodiments of the present invention may include one or more of the following operations, features, characteristics and / or advantages: (i) a means of tracking by a delayed detection framework; (ii) a tracker update workflow; and / or (iii) keyframe-based tracker parameter calibration based on detection results.

[0052] Some potential applications of the present invention may include: (i) artificial intelligence vision by accelerating object detection speed and response, and completing prototypes; and (ii) tracking athletes during sports competitions.

[0053] Some embodiments of the invention can include one or more of the following operations, features, characteristics and / or merits: (i) accurate, automatic-calibrated, real-time tracking; (ii) tracker automatic calibration with delayed object detection; (iii) tracking by delayed detection means, tracker module takes input frames and outputs directly, while sampled frames are fed into object detection module; (iv) outputs are used to calibrate tracker; (v) tracker update workflow with delayed detection outputs; and / or (vi) keyframe-based tracker calibration.

[0054] Some embodiments of the invention can include one or more of the following operations, features, characteristics and / or merits: (i) object tracking utilizes both object detection and traditional object tracking algorithms; (ii) delay is independent of detection algorithm; (iii) more complex but high-precision object detection CNNs (convolutional neural networks) can be used to improve overall precision; (iv) “keyframes” that speed up object tracking are used with different object tracking; (v) delayed object detection results can be used to calibrate tracking results in order to achieve real-time tracking; (vi) object detection and tracking are not in sequence; (vii) keyframe-based calibration methods are used to speed up tracking; (viii) independent of specific estimation algorithms like KLT (Kanade-Lucas-Tomasi); (ix) all traditional tracking algorithms (such as KCF (Kernelized Correlation Filter)) can be used to estimate current tracking position by keyframe-based tracker calibration; (x) system diagram is used to show algorithms working in real video recognition systems;

[0055] (xi) utilizes one detector and one tracker, where the output of the tracker is not fed back to the detector, but directly as system output; (xii) focuses on read-time problem tracking; and / or (xiii) focuses on real-time system object detection and object tracking combination.

[0056] Some embodiments of the invention can include one or more of the following operations, features, characteristics and / or merits: (i) hidden detection delay; (ii) uses output of tracker as system output; (iii) uses lagging detection results and keyframe-based mechanism to update parameters of tracker; (iv) does not wait for detection results; (v) keyframes are used to speed up parameter update of tracker; (vi) these keyframes guide the tracker to make more accurate output predictions of the object position, rather than using keyframes to reduce detection frequency; and / or (vii) KCF can be used as object tracking algorithm; (viii) can be used with tracking algorithms other than KCF.

[0057] IV. DEFINITIONS

[0058] Invention: The subject matter described by the term "invention" should not be construed as a literal instruction to cover each and every embodiment falling within the description of the term "invention" by the claims as filed or by the claims as ultimately issued following prosecution of the patent application; while the term "invention" is used to help the reader obtain a general sense of the disclosure, the disclosure herein is believed to be potentially new as indicated by the use of the term "invention" is provisional and temporary and subject to change during prosecution of the patent application as relevant information is developed and as the claims are potentially modified.

[0059] Embodiment: See the definition of "invention" above - similar caveats apply to the term "embodiment".

[0060] And / or: Includes or; for example, A, B "and / or" C means that at least one of A or B or C is true and applicable.

[0061] Comprise / comprising: Unless otherwise specified, means "including but not limited to".

[0062] Module / sub-module: Any hardware, firmware and / or software group operable to accomplish a certain function, regardless of whether the module is: (i) in a single local vicinity; (ii) distributed over a wide area; (iii) located in a single contiguous area within a larger piece of software code; (iv) located within a single piece of software code; (v) located in a single storage device, memory or medium;

[0063] (vi) mechanically connected; (vii) electrically connected; and / or (viii) data communication connected.

[0064] Computer: Any device having significant data processing and / or machine readable instruction reading capability, including but not limited to: desktop computers, mainframe computers, laptop computers, field programmable gate array (FPGA) based devices, smart phones, personal digital assistants (PDAs), body-worn or inserted computers, embedded device type computers, application specific integrated circuit (ASIC) based devices.

Claims

1. A computer-implemented object detection method, comprising: Receive multiple video frames in sequence and time sequence from video images including moving objects; Perform object detection on the multiple video frames to determine the identification of the moving object; During the object detection process, object tracking calibration is performed on the multiple video frames using bounding boxes and keyframes for the moving objects present in the multiple video frames. as well as During object detection and subsequent object tracking calibration, object tracking is performed to locate the moving object in at least some of the video frames to obtain the object tracking position of the moving object. The object tracking calibration and object tracking performed during object detection result in delayed detection results that conceal the detection delay associated with performing object detection on the plurality of video frames.

2. The object detection method according to claim 1, further comprising: Artificial intelligence vision is performed by utilizing the accelerated object detection speed obtained by using the delayed detection results to hide the detection delay.

3. The object detection method according to claim 1, wherein: The multiple video frames represent a sports competition; and The moving object is an athlete who moves during a sporting competition.

4. The object detection method according to claim 1, wherein: Performing object tracking calibration involves determining the position of the moving object by detecting the bounding box.

5. The object detection method according to claim 1, further comprising: Determine keyframes from the plurality of video frames, wherein the keyframes are frames where the position of the moving object changes relatively significantly; and Performing object detection includes applying an object detection algorithm to at least some of the plurality of video frames to locate the moving object in the video frames.

6. The object detection method according to claim 1, wherein, The sampled frame object position used as the object tracking position for a given video frame is the delay detection result.

7. An object detection system, comprising means adapted to perform all steps of the method according to any of the preceding method claims.

8. A computer program product comprising a computer program including instructions for performing all steps of the method according to any of the preceding claims when the computer program is executed on a computer system.

Images